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DOCUMENT -IDENTIFIER: US 5828370 A 

** See image for Certificate of Correction ** 

TITLE: Video delivery system and method for displaying indexing slider bar on the 
subscriber video screen 

Brief Surnmary Text (5) : 

Full -motion digital video requires a large amount of storage and data transfer 
bandwidth. Thus, video-on-demand systems use various types of video compression 
algorithms to reduce the amount of necessary storage and data transfer bandwidth. In 
general, different video compression methods exist for still graphic imagp*fi and for 
full -motion video. Video compression methods for still graphic imageg or single 
video frames are referred to as intraframe compression methods, and compression 
methods for motion video are referred to as interframe compression methods. 

Brie f Siimma ry Tf^-yt (6) : 

Examples of video data compression for still graphic images are RLE (run- length 
encoding) and JPEC^ (Joint Photographic Experts Group) compression. The RLE 
compression method operates by testing for duplicated pixels in a single line of the 
bit map and storing the number of consecutive duplicate pixels rather than the data 
for the pixel itself. JPEG compression is a group of related standards that provide 
either lossless (no image quality degradation) or lossy (imperceptible to severe 
degradation) compression types. Although lTPK^ compression was originally designed 
for the compression of still images rather than video, JPEH compression is used in 
some motion video applications . 

Brief Summary TRxt (7) ; 

In contrast to compression algorithms for still images , most video compression 
algorithms are designed to compress full motion video. Video compression algorithms 
for motion video use a concept referred to as interframe compression, which involves 
storing only the differences between successive frames in the data file. Interframe 
compression stores the entire image of a key frame or reference frame, generally in 
a moderately compressed format. Successive frames are compared with the key frame, 
and only the differences between the key frame and the successive frames are stored. 
Periodically, such as when new scenes are displayed, new key frames are stored, and 
SLibsequent comparisons begin from this new reference point. It is noted that the 
interframe compression ratio may be kept constant while varying the video quality. 
Alternatively, interframe compression ratios may be content -dependent, i.e., if the 
video clip being compressed includes many abrupt scene transitions from one -imagp* to 
another, the compression is less efficient. Examples of video compression which use 
an interframe compression technique are MPEG, DVI and Indeo, among others. 

Brief Summary Tf^yt ( 9) : 

A compression standard referred to as MPEG (Moving P-i ri-n-rp^fl Experts Group) 
compression is a set of methods for compression and decompression of full motion 
video images which uses the interframe compression technique described above. MPEG 
compression uses both motion compensation and discrete cosine transform (DCT) 
processes and can yield compression ratios of more than 200:1. 

Brief Summary Text (11) : 

An MPEG stream includes three types of pictures, referred to as the Intra (I) frame, 
the Predicted (P) frame, and the Bi-directional Interpolated (B) frame. The I or 
Intra frames contain the video data for the entire frame of video and are typically 
placed every 10 to 15 frames. Intra frames provide entry points into the file for 
random access, and are generally only moderately compressed. Predicted frames are 
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encoded with reference past frame, i.e., a prior In^^B frame or Predicted 

frame. Thus P frames only include changes relative to prior I or P frames. In 
general. Predicted frames receive a fairly high amount of compression and are used 
as references for future Predicted frames. Thus, both I and P frames are used as 
references for subsequent frames. Bi-directional p-i r^n•rf>fi include the greatest 
amount of compression and require both a past and a future reference in order to be 
encoded. Bi-directional frames are not used for references for other frames. 



Brief Summary Text (12) : 

After the I frames have been created, the MPEG encoder divides each I frame into a 
grid of 16=16 pixel squares called macro blocks. The respective I frame is divided 
into macro blocks in order to perform motion compensation. Each of the subsequent 
pictures after the I frame are also divided into these same macro blocks. The 
encoder then searches for an exact, or near exact, match between the reference 
picture macro block and those in succeeding pictiirea. When a match is found, the 
encoder transmits a vector movement code or motion vector. The vector movement code 
or motion vector only includes information on the difference between the reference 
frame and the respective succeeding pirtura . The blocks in succeeding pH rturf^g that 
have no change relative to the block in the reference pi r^_u^rf> or frame are ignored. 
In general, for the frame (s) following a reference frame, i.e., P and B frames that 
follow a reference I or P frame, only small portions of these frames are different 
from the corresponding portions of the respective reference frame. Thus, for these 
frames, only the differences are captured, compressed and stored. Thus the amount of 
data that is actually stored for these frames is significantly reduced. 

Brif^f Summary Tpyt: (14) : 

Each picture or frame also includes a picture header which identifies the frame and 
includes information for that frame. The MPEG standard also includes sequence 
headers which identify the start of a video sequence. Sequence headers are only 
required once before the beginning of a video sequence. However, the MPEG-2 standard 
allows a sequence header to be transferred before any I frame or P frame. The 
sequence header includes information relevant to the video sequence, including the 
frame rate and picture size, among other information. 

Brief Summary Text (23) : 

In the preferred embodiment, the media server stores normal play streams, and may 
store one or more corresponding trick play streams, i.e., fast forward and/or fast 
reverse video streams, for each of the normal play streams. In the preferred 
embodiment, the media server stores normal play video streams which are preferably 
compressed using any of various types of video compression methods, preferably an 
MPEG method. The system of the present invention generates an index table or look-up 
table for each normal play stream which enables indexing to different positions in 
the stream. In creating an index table for a normal play stream, the system first 
analyzes the normal play stream and preferably creates a normal play time standard 
based on presentation timestamps comprised in the normal play stream. The system 
then preferably creates an index table or look-up table for the normal play stream 
using the normal play time standard. Each index table comprises an array of 
two- tuples, wherein the two- tuples are the normal play time standard and an index or 
offset into the respective stream. The index tables enable indexing to different 
locations in the stream. If the media server stores trick play streams, the system 
also preferably creates index tables for the trick play streams which enable 
indexing between the normal play and trick play streams. 

Brief Summary Text (26) : 

The media server then uses the respective index table to index into the normal play 
video stream. After the media server calculates the new normal play time, the media 
server uses the calculated normal play time to retrieve the appropriate nf f P!e^. in 
the normal play stream index table. This offset: is then used to begin play of the 
normal play stream at the retrieved offset. The set top box then preferably adjust 
the location of the knob on the slider bar to reflect the correct position of the 
video stream being output. The media server may also optionally examine the current 
normal play time and offset of the normal play stream being output in order to halt 
the normal play stream at an appropriate point before outputting the stream at the 
retrieved Q££a£t. 



Brief Summfiry Text (27) : 

In one embodiment, when the user first manipulates the slider bar to jump to a new 
location in a video stream, the media server caches the current normal play time and 
byte offset of the normal play stream. Thus, if the user later desires to return to 
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the original point in the" movie before any slider bar adjrretments were made, the 
user presses a key on the remote control or selects a return option on the slider 
bar. This selection causes the media server to retrieve the appropriate normal play 
time and offset in the normal play stream index table. This of f Rp^t is then used to 
being play of the normal play stream at the location where the stream originally was 
prior to any slider bar adjustments. 

Detailed Deficription Text (6) : 

The normal play streams are preferably compressed video streams. The present 
invention operates independently of the type or format of the video streams. Thus 
the normal play video streams may be compressed in any of various types of formats, 
including MPEG-1, MPEG- 2, Motion iTPKrt, QuickTime, etc. Further, the present 
invention operates independently of the frame rate and other presentation 
characteristics . 

Df^tailf^d Df^arriptinn Tp>yt (8) : 

Referring now to FIG. 2, a diagram illustrating a user viewing a subscriber unit 52 
according to the preferred embodiment of the present invention is shown. As shown, 
the subscriber unit 52 comprises a television or other display device 53 . The 
television 53 includes a display screen 59 for displaying video images , as is well 
know in the art. It is noted that the television or display device 53 may comprise 
any various types of display units, including a digital TV or high definition TV 
(HDTV), a computer system having a video monitor, or other types of display devices. 



rtPl-ailpti DPRnr-ipt-i nn Text (24) : 

As discussed above, an MPEG encoded stream includes a plurality of I- frames which 
are intracoded pi r^n-rea, and a plurality of B and P frames which are interceded 
frames. The I frames each contain video data for an entire frame of video and are 
placed periodically in the sequence. The P and B frames include change information 
relative to prior or subsequent frames. Each p-i rtii-rp; or frame also includes a 
pint II re header which identifies the frame and includes information for that frame. 
An MPEG encoded stream further includes one or more sequence headers which include 
certain information regarding the video sequence, including the frame rate and the 
pi rture size, among other information. The sequence headers include presentation 
timestamps which indicate the play time of the video sequence. 

Detailed DeRr-ri pt -i nn Tevt (25) : 

In step 106 the system of the present invention preferably maps the presentation 
timestamps to a "normal play time" standard. Thus the system of the present 
invention defines a multimedia index based on the concept of "normal play time" that 
can be associated with a "position" within a multimedia title. The normal play time 
standard is used for indexing to different positions or locations in the normal play 
stream according to the present invention. The normal play time standard may also be 
used for indexing between streams having different presentation rates, i.e., for 
indexing between normal play and trick play streams . With regard to trick play 
streams, positions are defined to be equivalent between normal play and/or trick 
play video streams having different presentation rates when the content present at 
the respective position is conceptually substantially equivalent, i.e., when the 
same or siibstantially the same image in the sequence is being presented, allowing 
for differences in resolution and other encoding parameters that may be particular 
to the stream. 

Detailed Description Text (28) : 

It is noted that other methods may be used to generate index tables for the normal 
play and trick play streams. In an alternate embodiment, the system of the present 
invention creates index look-up tables using a list of offaetg ho each mp (n-ronp of 
Eicture^) . In another embodiment, the system uses the presentation timestamps in the 
video stream for indexing. Other methods for creating the index tables may be used, 
as desired. It is also noted that other methods may be used to generate normal play 
time (NPT) standard index tables for the normal play stream 

Detailed Description Text (29) : 

In step 108 the system creates an index look-up table for the respective multimedia 
normal play stream. The index look-up table for the normal play multimedia stream 
comprises an index or array_ of two- tuples , wherein each tuple- comprises a normal 
play time value and an offset in the respective stream. If trick play streams exist 
in the system in step 110, the system optionally creates index look-up tables for 
the trick play streams in step 112. The index look-up tables for a normal play, fast 
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forward, and fast revers^ stream are shown in FIG. 6. As^nown, each tuple comprises 
a normal play time value and a corresponding file of f Pif='t within the stream. For the 
normal play stream, the normal play time entries comprise the normal play time 
values computed in step 106. For trick play streams which have constant bit rate 
encodings, a scale factor is introduced into the normal play time values of the 
index look-up tables to compensate for the different presentation rates. 

De^tailf^d Dg^gririptiQn Text (30) : 

The index look-up tables specify indices or entries each based on a normal play time 
and a file of f sf^t to allow the multimedia server 50 to initiate or stop play at a 
particular normal play time point in the multimedia stream. The index look-up table 
for the normal play stream is used according to the present invention to enable the 
media server 50 to output video data for the normal play video stream at various 
locations or positions as indicated by user input to the slider bar 54 . The index 
look-up tables for the trick play streams also allow the multimedia server 50 to 
transfer to and between equivalent positions of streams having different 
presentation rates, i.e., between normal play and trick play streams. 

DfttflllPid Pfifif:ription Text (31) : 

In one embodiment, the index look-up table only includes tuples representing valid 
positions for starting, stopping, or transferring between the streams. It is noted 
that equivalent positions in multimedia streams having different presentation rates 
will have equal NPT values, although the actual time of presentation from the 
beginning of the stream to that position will differ for the different streams. It 
is also noted that equivalent positions in multimedia streams having different 
presentation rates, although having equal NPT values, will have different byte 
nf f gf^t:g due to a presumptive difference in length of the streams having different 
presentation rates . 

Detailed Description Text (32) : 

The creation of the look-up tables is independent of any particular type of video 
compression or MPEG representation. In the preferred embodiment where MPEG 
compression is used, the index look-up tables are created by scanning through the 
MPEG file, noting random access points in the MPEG file, and converting from the 
presentation timestamp in the MPEG file into the normal play time standard. 
Conceptually, each index table comprises an array of normal play time vs. scenes, 
and any particular imagf> or frame in the movie can be identified by the normal play 
time value. As noted above, an index table is created for the normal play stream as 
well as any trick play streams, e.g., fast forward and fast reverse streams. Each of 
the of f ap-ta stored in the index table is an index from the normal play time to a 
byte of f gg^t in that MPEG file vAtere the particular scene begins. 

Detailed Description Text (33) : 

Therefore, in the preferred embodiment, the system of the present invention uses a 
normal play time standard for the index tables. In an alternate embodiment, the 
system of the present invention creates index look-up tables using a list of of f gf^tfl 
tn parh any (nronp nf p-i nhnTPfl) _ As noted above, other methods for creating the 
index tables may be used, as desired. 

Detailed Description Text (43) : 

After a new normal play time is determined based on the received user input in step 
204, in step 206 the media server 50 indexes to a new position in the normal play 
stream based on the normal play time determined in step 204. Thus in step 206 the 
media server uses the normal play time calculated in step 204 to retrieve an of f gf^t. 
from the normal play stream index table. In step 208 the media server 50 outputs the 
normal play stream at the indexed location or offset position determined in step 
206. 

Detailed Dftscription Text (45) : 

Thus, a change in position or location of the output of a respective stream being 
output at a given normal play time is accomplished by finding the tuple in the 
respective index table for the nearest normal play time and utilizing the associated 
file of f piph as the point to initiate play of the stream. 

Detailed Description Text (47) : 

Therefore, the present invention comprises a system and method -which provides"a 
graphical icon or si ider'bar for "indexing to different positions or locations in a 
normal play video stream. The present invention examines the presentation times tamps 
in the sequence headers of the normal play stream and creates a normal play time 
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standard. The system thefr creates an index table or look^^ table for the stream. 
The index table for the stream comprises normal play time values and corresponding 
of f fie t a into the respective stream. During play, when the user manipulates or 
adjusts the slider bar knob, the subscriber device or set top box calculates a 
corresponding value and provides this value to the media server 50. The media server 
50 uses the respective index table to intelligently jump or index to the appropriate 
position in the normal play stream and begin outputting the stream at this new 
position. 

CLAIMS : 

4. The method of claim 1, wherein the media server stores an index table 
corresponding to the video stream, wherein the index table comprising a plurality of 
tuples, wherein each of said tuples comprises a first value and a second offset 
value which is an offset into a position of the video stream, wherein said media 
server determining a new position of the video stream in response to said received 
user input comprises the media server indexing into the index look-up table to 
obtain an nffflph intn ^bp vidpo stream. 

5. The method of claim 4, wherein each of said plurality of tuples comprises a 
normal play time value and an off apt inho the vidf^o stream; 

wherein said subscriber device providing information comprises the subscriber device 
providing a value within a predetermined range indicating a relative position within 
the video stream based on said received user input; and 

wherein said media server determining a new output position for the video stream 
comprises the media server determining a normal play time value based on said 
received value in said predetermined range. 

16. The interactive video delivery system of claim 15, wherein the media server 
further comprises: 

a memory which stores an index table corresponding to the video stream, wherein the 
index table comprising a plurality of tuples, wherein each of said tuples comprises 
a first value and a second n-Ff af^h value which is an of f fif^^- into a position of the 
video stream; 

wherein said means for determining a new output position for the video stream 
indexes into the index look-up table stored in said memory to obtain a n offa^i- ^r^^o 
hhf^ vidRo stream. 

17. The interactive video delivery system of claim 16, wherein each of said 
plurality of tuples comprises a normal play time value and an offaf^t: into the v-idpo 
stream; 

wherein said means for providing information provides a value within a predetermined 
range indicating a relative position within the video stream based on said received 
user input; and 

wherein said means for determining a new output position for the video stream 
determines a normal play time value based on said received value in said 
predetermined range. 

25. The method of claim 24, wherein the media server stores an index table 
corresponding to the video stream, wherein the index table comprising a plurality of 
tuples, wherein each of said tuples comprises a first value and a second offset 
value which is an offset into a position of the video stream, wherein said media 
server determining a new output position of the video stream in response to said 
received user input comprises the media server indexing into the index look-up table 
to obtain an offset into t:he vidp^o stream. 

27. The interactive video delivery system of claim 26, wherein the media server 
further comprises : 

a memory which stores an index table corresponding to the video- stream,- wherein tKe 
index table comprising a plurality of tuples, wherein each of said tuples comprises 
a first value and a second offset value which is an offset- into a position of the 
video stream; 
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wherein said means for determining a new output position for the video stream 
indexes into the index look-up table stored in said memory to obtain a n nffRPt inhr. 
the video stream. 
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AH.QM-ar-^ Tpyh (1) : 

Disclosed is an aesthetic imaging system (20) for use in editing digital image s. The 
aesthetic imaging system includes an imaging program (21) that runs on a personal 
computer (28) having an image capture board (30), a monitor (32), a video source 
(34) for providing digit al_iiiiagea to be edited by the aesthetic imaging system, and 
a pen and tablet 38 for use in 

Abstract Text (2) : 

editing the images . The imaging program includes a unique combination draw tool that 
includes a freehand draw mode, a curve mode and an undo mode that are available 
without cycling through menus. The combination draw tool may be used with any of the 
draw tools. Another feature of the imaging program is autoblend, a rectangular user 
interface that is invoked by each of the shape tools. The autoblend interface 
simplifies editing when using shape tools by consolidating the move, paste and 
blend, and paste without blending commands into a single, convenient interface. 

Brief Summary Text (5) : 

People are with increasing frequency consulting physicians about cosmetic surgery. 
While in many cases the patients considering cosmetic surgery have an impressive 
understanding of the procedures available and medical terms used to describe these 
procedures, it is apparent that the slightest miscommunication may result in dire 
consequences. This has promoted the use of computer imaging to facilitate 
communication between the physician and prospective patient. Specifically, high-end 
aesthetic imaging systems allow a physician to take pre-operative digital i mage s of 
the patient, e.g., including profile and frontal views. The images are stored in 
memory in the computer where they can then be edited. Using feedback from the 
patient, the edited -iniagf^Pt are useful in fully understanding the procedures desired. 
The visual support provided by a computer-based imaging system is extremely valuable 
on both sides. A cosmetic surgeon can more readily understand what patients hope to 
achieve by a cosmetic surgical procedure; and patients can view a detailed visual 
representation of predicted results, including both the benefits and limitations of 
the procedure. 

Brief Summary Text (6) : 

Existing aesthetic imaging systems typically use a number of tools to allow a 
physician to manipulate a pre-operative image of a patient to illustrate an intended 
post -operative result. Prefeirably, the tools allow the physician to manipulate the 
preoperative -imagf^ during a consultation with a patient. By manipulating the image 
with the patient in attendance, the patient receives immediate feedback from the 
displayed results. For a successful preoperative consultation, the use of the 
editing tools should be as unobtrusive as possible. During the consultation, a 
physician would like the patient to focus on the end results of the surgery, rather 
than the technologic wizardry used to demonstrate those results on the video 
monitor. Unfortunately, the editing tools used in existing aesthetic imaging systems 
typically hinder rather than help the physician in demonstrating the results that 
may be achieved through cosmetic surgery. 

Brief Summary Text (7) : 

Among others, a disadvantage of existing aesthetic imaging programs is that a 
physician or facilitator in a pre-operative consultation typically must go back and 
forth through many windows-based menus in order to edit an image . Cycling between 
the various menus to invoke the tools necessary for a consultation is 
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disadvantageous in that it is time consuming. For example, some physicians schedule 
a follow-up visits for patients to allow the physician time to edit the images. More 
important, however, is that the process is distracting to the patient and tends to 
make the pre-operative consultation all the more mystifying. As a result of the 
disadvantages associated with prior art systems, some patients lose interest or 
become frustrated with the interview, both of which may reflect back on the 
physician. 

Rrlfif Summary Text (8) : 

A further disadvantage of existing aesthetic imaging systems is that it is 
impossible for a physician or facilitator to display different combinations of the 
edits that they have performed. In existing aesthetic imaging programs, as a 
physician edits a patient's image , the physician's edits are added to the 
preexisting edits of the image . Most programs are only capable of showing two 
version of the patient's i mage ; the imedited, original version, and the final edited 
version incorporating all of the physician's changes. It is therefore difficult for 
the physician to show various combinations of the edits that had been performed. For 
example, a physician may edit an image to remove wrinkles around a patient's eyes 
and to narrow the patient's nose. Existing aesthetic imaging programs only allowed 
the physician to simultaneously show all of these changes. If, for example, the 
patient wished to view the changes to the eyes without the changes to the nose, it 
was difficult for the physician to easily separate out the sequence of edits that 
had been performed to adjust the patient's nose. An improved aesthetic imaging 
system in which a physician can more easily edit pre-operative imageR in response to 
a patient ' s suggestions and inquiries would be extremely advantageous . 

Brief Summary Text (9) : 

A Still further disadvantage of existing aesthetic imaging systems is that the 
systems allow a physician to perform nearly flawless editing of a patient's image , 
Unfortunately, the edits performed by a physician on an aesthetic imaging system are 
often unobtainable results that cannot be achieved when actual surgery is performed. 
Unless the physician is especially skilled at using the aesthetic imaging system, it 
is difficult to show the patient achievable results, which typically fall within a 
range somewhere between the original patient image, and the optimum results as 
displayed by the edited image on the screen. It therefore would be advantageous to 
develop an aesthetic imaging system that allowed a physician to display more 
realistic results that are achievable through surgery. 

Brief Summary Text (11) : 

The invention is an aesthetic imaging system for use in editing digital images. The 
aesthetic imaging system includes a unique user interface that allows edits to be 
performed more efficiently and with less confusion to the patient. 

Brief Summary Tevh (12): 

In one aspect of the invention, a method of editing a digital image comprised of a 
plurality of color pixels in an aesthetic imaging system is disclosed. The aesthetic 
imaging system including a processor, a memory, a monitor, and a pen and cooperating 
tablet for controlling a cursor displayed on the monitor. The pen has a depressable 
tip and a side button, each of which include an on status and an off status, wherein 
the position of the pen tip relative to the tablet determines the position of the 
cursor on the monitor. The method comprises: (a) evaluating the following variables: 
(i) the status of the tip of the pen; (ii) the status of the side button on the pen; 
and (iii) movement of the pen tip relative to the tablet; (b) actuating a freehand 
drawing mode if a first set of variables are present, wherein movement of the pen 
relative to the tablet edits pixels that are located at positions corresponding to 
the position of the cursor; and (c) actuating a curve drawing mode if a second set 
of variables are present, wherein a line segment is displayed between two endpoints 
and movement of the pen relative to the tablet stretches the line segment, forming a 
curve and editing pixels that are located at positions corresponding to the position 
of the curve. 

Brief Summary Text (16) : 

In another aspect of the invention, an improved prioritize feature is described for 
viewing an image . A user may identify several areas in a modified patient image 
containing edits that alter the image from the original image . As each area is 
identified by the user, an identifying tag is assigned to each of the areas. When 
desiring to show various combinations of the edits that have been performed on the 
image ^ the user may select the areas to display using the identifying tags. A user 
may therefore quickly cycle through various permutations of the procedures that have 



2 of 15 



7/29/03 5:32 P^ 



been edited for patient display. 



RT-if>f fiiimmary Tp^yf (17) : 

In Still another aspect of the invention, an improved user interface is provided to 
minimize the distraction of a patient as the patient is watching the imagpi being 
edited. Preferably, a menu bar on the top of the display is removed during most 
editing, so that only the imagf^, of the patient is displayed. When the menu bar must 
be displayed, the bar itself is transparent to allow the patient to see the 
image-through the menu bar. Only the commands and the outline of the menu bar are 
presented in a contrasting color, minimizing the overall visual impression created 
by the menu bar. 

Rrifif Nummary T^xt (18) : 

In yet another aspect of the invention, a warp tool is described that allows a user 
to quickly and easily manipulate various features in an image , To use the warp tool, 
a user first encircles a portion of the -image to be edited. Once the area has been 
selected, the user may tip the pen to designate a stretch point within the selected 
area. As the user floats the pen over the tablet, the image is then stretched as if 
pulled from the stretch point. Areas of the image in the direction of stretch are 
compressed, and areas away from the direction of stretch are expanded. Areas 
surrounding the warping area are automatically adjusted to ensure that there is no 
discontinuities with the warping area. The manipulation of the image is performed in 
real-time, allowing a patient to see the warping as it is being performed by a user. 



Rrifif Summary Text (19) : 

An advantage of the tools and features described herein are that they improve the 
overall experience of a patient during a preoperative visit with a physician. The 
powerful tools in the aesthetic imaging system allow the physician to easily 
manipulate the patient ' s image in response to feedback provided by the patient . The 
aesthetic imaging system interface also allows the patient to focus on the Imagft 
being manipulated, rather than on the aspects of the aesthetic imaging system that 
allow the manipulation. The end result is an improved preoperative visit that 
provides a more realistic impression of the results that a physician may achieve 
through surgery. 

Drawing Dftacription Text (5) : 

FIG. 3 is a flow chart illustrating an exemplary routine by which digital imagfts may 
be viewed and edited using the aesthetic imaging system; 

Drawing Description Text (7) : 

FIG. 4B is a flow chart of an exemplary routine for calculating a checksum value and 
comparing the calculated value to a previously stored value to determine if an image 
has been altered; 

Drawing DFiRcriptiQn Text (9) : 

FIG. 6 is a pictorial representation of an image to be edited; 
Drawing Deacripti nn Texh (10) : 

FIGS. 7A-7E are pictorial representations of editing an image using a prior art 
imaging program; 

Drawing Deflnription Text (19) : 

FIG. 13 is a flow chart of an exemplary routine for viewing imageg in accordance 
with the invention; 

Drawing Defirript-i nn Tex^ (21) : 

FIGS. 15A-15C illustrate a split image option of viewing images in accordance with 
the invention; 

Drawing Description Text (22) : 

FIG, 16 is a pictorial representation illustrating the use of a translucent image to 
allow a patient to accurately position themselves in order to capture a second image 
having the same location and orientation as an original stored image ; 

Drawing Desnripti on Text (23) : 

FIG. 17 is a pictorial representation illustrating a compare image wherein a 
presurgical image of a patient is compared side-by-side with a postsurgical image 
having the same location and orientation; 
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n-raw-ing Dp^Rrr-jph-t nn Tf>y^ (24) : 

FIGS. 18A-18C are pictorial representations illustrating the use of a warp shape 
tool to edit a patient » s_jjnag£ 

Drawing Dp^Rrript inn T^yt (26) : 

FIG. 20 is a pictorial representation illustrating the use of a transparent menu bar 
when viewing a n imagp^ of a patient; and 

Df^tailf^d Description Text (4) : 

The aesthetic imaging system also includes an image capture board 30 that is coupled 
to the processing unit 22, a monitor 32, video source 34, and printer 36. The video 
source, monitor, and printer are coupled to the processing unit 22 through the imagf! 
capture board 30. The video source may include one or more video cameras, a VCR, a 
scanner, or sinilar source for providing digital images to be edited by the 
aesthetic imaging system. The aesthetic imaging system further includes a pointing 
device, which is preferably a stylus (pen) and tablet 38, that is connected to the 
processing unit 22. In addition, the aesthetic imaging system may include a modem 40 
to provide on-line capabilities to users of the system, such as technical support 
and teleconferencing. 

Dfttailed Dftscription Text (5) : 

The -imagp> capture board 30 has a plurality of buffers in high-speed memory, e.g., 
RAM, that are used by the imaging program 21 to provide very fast response times to 
image edits. With reference to FIG. 2, four buffers are illustrated for use in 
explaining the operation of the aesthetic imaging system. These include an original 
image buffer 50, a modified image buffer 52, a current image buffer 54, and a 
working buffer 56. Suitable -image capture boards for use in the aesthetic imaging 
system include the Targa +64 and Targa 2000 boards, distributed by Truevision, Inc. 
of Indianapolis, Ind. The buffers are discussed in regard to a single pose only, 
such as a profile or front view of a person. 

Detailed Deariription Tex:t (6) : 

The original image buffer 50 contains an unedited digital image, for example, a side 
profile picture of a potential patient. The modified image buffer 52 contains any 
edits made to a copy of the original -image . The modified image buffer is updated 
during a save and after each session. The current image buffer 54 contains 
information identical to the modified image buffer upon beginning a session. 
Thereafter, edits made to the current i mage are saved in the working buffer 56 as an 
overlay to the current image . During a save, the contents of the current image 
buffer 54 are copied to the modified image buffer 52, and the working buffer 56 is 
cleared. 

Detailed Desnription Text { 8) : 

Tmagft 

Detailed Description Text (9) : 

A digital photograph or picture of a patient. 

Detailfid Deficription Text (11) : 

The "pen" that may be used to select menus, modify images, and carry out other 
commands in the program. The stylus controls the cursor, just as a mouse pointing 
device does on a personal computer. 

Detailed Description Text (23) : 

Pressing the tip of the pen, then dragging it across the tablet while maintaining 
pressure. When using drawing and shaping tools, this turns the cursor into a drawing 
tool, enabling the user to draw freehand objects or "brush" the image in any manner. 



Detailed Description Text (25) : 

At block 64, a test is made to determine if the Camera option has been selected from 
the main menu, indicating that the user wants to take a picture of a patient. If the 
Camera option has been selected, a routine to implement this command is called at 
block 66. A suitable subroutine for this task in illustrated in FIG. 4. Upon return 
from the Camera routine, the program loops to block 62 . 

Detailed Deflcri pt i nn Text (26) : 

If the Camera option was not selected, a test is made at block 68 to determine if 
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the Storage option has been selected for the main menu, indicating that the user 
wants to load a n -Imag f^ (pi ^^^^1•rp>> from computer storage, e.g., a hard drive. If the 
Storage option was selected, the image (s) to be edited or viewed during the current 
session are selected at block 70. 

Dfttailftd Dfifinriptinn Text (27) : 

At block 72, the selected^images auce copied to the appropriate buffers in the frame 
grabbing board, as described in FIG. 2 and the accompanying text. For example, if 
the selected-image is an original -imagf^ that has not yet been edited, the original 
imagp^ will be copied to the original, modified, and current image buffers. If the 
selected imagf^ ig an -imagfi that has previously been modified, the original imagpi is 
copied to the original -imagf^ buffer 50 and the modified image is copied to both the 
modified and current imagf* buffers 52 and 54. It will be appreciated that the number 
of 1 magf^R that may be loaded at one time will be limited, in part, by the capacity 
of the frame grabbing board. The program then loops to block 62. 

Dfiha-ilf^d Dp Rr?r i pt ion Text (30): 

If the View option has been selected, a view subroutine is invoked at block 84. An 
appropriate routine for the View option is shown in FIG, 13. The program then loops 
to block 62. At block 86 a test is made to determine whether the Exit option has 
been selected from the main menu. If not, the program loops to block 62. Otherwise, 
any edits to the image* are saved at block 88. At this point in the program, the 
imagp^ in the current image buffer 54 is saved to the modified image buffer 52, and 
the working buffer 56 is cleared. Tlie program then terminates. 

Detailed Description Text (31) : 

Taking P-iri-nrpfl of a Patient Using an Inverse Image 
Detailed Description Text (32) : 

FIG. 4A illustrates an exemplary user interface that utilizes a video camera for 
acquiring a digit al^Lmage of a patient. It is noted a scanner or other input device 
may also be used to input an image into the aesthetic imaging system. In FIG. 4A, 
the solid blocks indicate user interface options presented to the user by the 
aesthetic imaging system and the dashed blocks represent system responses to the 
decisions made. At block 100, a patient is positioned in front of the video-camera. 
In a preferred embodiment of the aesthetic imaging system, an inverse or "mirror" 
image of the patient's image will be displayed on the monitor, as indicated at block 
101. The inverse image is computed using data from the original image , and is 
representative of how patients see themselves day to day when looking into a mirror. 
More specifically, digital images are comprised of pixels or pirtnre elements. It is 
known to those skilled in the art how digital image pixels may be manipulated to 
create an -image that is the inverse of the original. 

Detailed Description Text (33): 

Displaying an inverse image of a patient is advantageous when taking pre- and 
post-surgical pir^t-n-refi of patients because it allows patients to more easily center 
or otherwise position themselves on the monitor. Without the pixel manipulation, the 
input from a camera or other digital device may create confusion when positioning a 
patient. Under normal viewing, if a patient appears left of center in the monitor, 
they are in reality too far to the right. In this instance, a typical patient's 
reaction is to move even further to the right. With a mirror image displayed, the 
tendency of most patients is to naturally adjust to the desired position. 

DehaileH negn-r-i i r>n Text: (34) : 

The displaying of a mirror_Junage is particularly important when taking post-surgical 
pictures. In post -surgical p-i rtii-reg^ it is advantageous to have the patient in 
exactly the position they were in when taking the pre-surgical picture. For 
pn.cti--Rn-rgi r;al pi rt-n-reg, the aesthetic imaging system will preferably display a 
translucent inverse image of the pre-surgical p-irhnre on the monitor, and then 
overlay an inverse image nf Mie p^r^n■re currently being taken. As shown in FIG. 16, 
a translucent patient image 370, in this case a patient's profile, is displayed on 
the monitor. The translucent image is the preoperative image taken prior to 
undergoing a surgical procedure. A "live" video image 372 of the patient is also 
displayed under the translucent inverse image . By moving the relative positions of 
the patient and the camera, a user may position the patient in the identical 
orientation with which they took the pre surgical picture. When the patient is 
appropriately positioned, the tip of the pen may be pressed anywhere on the tablet 
to freeze the digital image on the monitor. Patients can thus easily align 
themselves with their former pirt-nre to achieve very similar before cind after 
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p-i rtn-rfifl . 



nf>^a^np^H nPRrr^p^^nTl< Tf>v^ (35) : 

The advantage of the translucent method of aligning a patient is shown in the 
side-by-side display shown in FIG. 17. As shown in FIG. 17, the aesthetic imaging 
system of the presemt invention may generate a side-by-side display of the two 
imagf^fl to allow a patient to easily and accurately compare a presurgical picture* 
with a post-surgiccil piciuire. The left half of the monitor may display the 
presurgical j magp 370, and the right half of the monitor may display the 
post-surgical -imagA 372, Allowing a patient to view the two images side-by-side in 
precisely the same orientation provides the patient with an accurate impression of 
the results achieved by surgery. 

Dpra-ilF^H nppir'-r-ipt-irtn Tf^vt (36) : 

Upon establishing a desired position for the patient, the imagf^ is focused and sized 
at block 102 by using the aesthetic imaging system to adjust the electronic controls 
on the video camera. After any adjustments have been made to the camera, at block 
104 the tip of the pen is pressed anywhere on the tablet to freeze the digital image 
onto the monitor. At block 106, the user makes a determination if the image 
currently displayed on the monitor is acceptable. If the image is not acceptable, 
the routine loops to block 100. If the -imagp is acceptable, an appropriate command 
is entered at block 108 and the -imagp is stored in nonvolatile memory for future 
viewing . 

Detailed Description Text (37) : 

At block 110, a test is made to determine if an exit or other similar command has 
been entered by the user, i.e., if any more pinturf^R are to be taken. If additional 
p-intiireg are to be taken, the program loops to block 100. Otherwise, at block 111 a 
checksum value (described below) is calculated by the imaging program for each 
(original) -imagfi that has been stored. At block 112, the imaging program stores each 
image and its associated checksum value. The routine then returns to block 68 of 
FIG. 3. 

Df^tailp H nf>i=ir;rip1--ir>n Tpyh (38) : 

Determining Authenticity Using Checksum Values The checksum value is an addendum to 
an original stored_image that is used to determine its authenticity when the image 
is subsequently displayed or printed. Those skilled in the art will recognize that 
there are a number of methods of implementing such a checksum procedure. For 
example, one checksum computation is to add up the grayscale values for one of the 
colors, i.e., red, green, or blue, for each pixel comprising the image . Assuming a 
screen of 640 by 480 pixels and 256 colors per pixel, the checksum values would 
range from 0 to (640) (480) (255) . When an -imagp is recalled for display or to be 
printed, the checksum value is recalculated. If the image has not been modified, the 
newly calculated checksum value will be equivalent to the addendum value, and the 
image is certified as being unaltered. If the image has been modified, the checksum 
values will vary, indicating the image has been modified. In this instance, an 
indication of the fact that the -imagf^ has been modified may be displayed or printed 
with the image, if desired. 

Detailed Df^Rrriphinn Text (39) : 

The authentication of an original image using a checksum value is ideal for 
situations in which physicians display before and after pictures of a patient who 
has undergone cosmetic surgery. In some instances, viewers are skeptical as to 
whether an "after" -imagfi is really representative of a patient's appearance after 
surgery. This is in reaction to beliefs that post-surgical images have been altered 
to make the patients look better. For example, there may be temptation to slightly 
fade wrinkles or otherwise edit features that the physician was attempting to 
address in a surgery. 

nf^^a-^^pri Dfifim-iptinn Tf>yt (40) : 

Using the described checksum feature, the post-surgery image can be verified as an 
authentic, unaltered^image based upon the addendum value, and the veracity of the 
imagp is not questioned. This is beneficial to physicians when illustrating 
post-surgical results during lectures or in other teaching situations. 

Detailed Description Text (41) : 

FIG. 4B illustrates an exemplary routine for determining whether or not an original 
imagf^, i.e., pre- or post-surgical -imagfi, has been altered in accordance with the 
invention. This routine may be invoked whenever a pre- or post-surgical image is 
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displayed on a monitor or printed on a page. At block 114, a test is made to 
determine whether the imagfi to be displayed is portrayed as an "original image " that 
was not modified, e.g., a before or after picture. If the image is supposed to be an 
original image ^ a current checksum value for the image is calculated at block 115 . 
At block 116, the calculated checksum value for the image is compared to the 
checksum that was stored when the image was acquired by the aesthetic imaging 
system, e.g., when the picture was taken. 

np^a^1f^d Description Text (42) : 

At block 117, the calculated checksum value is compared to the stored checksum value 
to see if they are equivalent. If the two values are not equivalent, at block 118 an 
icon is added to the i magf^. , e.g., displayed or printed along with the image , 
indicating that the -imagR has been altered. If the checksum values are equivalent, 
an icon is added to the imagp at block 119 verifying the authenticity of the image. 
Once either icon has been added to the imagp, m r if thp imagfi being printed or 
displayed is not an original -imagp, the program terminates. As will be appreciated, 
the same checksum computational method must be used on each image, i.e., when an 
original imagf^ is acquired and when an image is to be displayed, or the comparison 
will be meaningless. 

nfatailed Df^firription Text (43) : 

While the use of a checksum is contemplated in the exemplary routine for determining 
whether an original_iinage has been altered, it will be appreciated that other 
techniques may be used to detect when an original digital image has been modified. 
For example, a flag or other marker may be uniquely associated with the original 
H-igit-al image . If the flag or Other marker is absent from the digital image being 
displayed, the digital image is a copy that is presumed to have been changed. 

Dehailef] neflrri pt i on Text (44) : 

Alternatively, two versions of the image may be stored, included an unaltered 
original and a copy. The two images may be compared in order to determine whether 
modifications have been made to the copy. 

Detailed Dean-ript ir>n Text (47) : 

A disadvantage of prior art aesthetic imaging systems is that a physician or 
facilitator may have to cycle back and forth between several menus in order to 
properly edit an image , In an effort to minimize the number of menus required, the 
imaging program includes a unique combination draw (CD) feature that generally works 
with all of the drawing tools. The CD feature allows a user to freehand draw, use 
curves to edit an image, as well as undo using either freehand or curves, without 
having to invoke a separate menu for each item. 

Detailed Descr iptinn Text (49) : 

To freehand draw, the user presses on the tablet with the pen tip and continues 
pressure while moving or "rubbing" the pen on the tablet. At this point the chosen 
color is written onto the image at the location on the monitor that corresponds to 
the pen location. Pressing the side bar while repeating the motion will allow the 
user to selectively remove any edits to the image using a freehand motion. 

Pet-ailed Deflr-ri pti nn Text (50) : 

To draw a curve, the user must set a first anchor point by selecting with the pen. 
Thereafter, as the user moves or "floats" the pen across the tablet, a green line 
will extend from the first anchor point to the current position of the pen. In a 
desired location, a second anchor point is set by selecting with the pen. Once both 
anchor points have been established, the green line appears on the monitor as a 
segment between the two anchor points. To create a curve, the user floats the pen 
across the tablet. The system will display a curved line bending and moving with the 
movement of the pen. The pen movement (top to bottom, or side to side) determines 

the arc of the curve. As the curve moves across the image , the image is edited in 

accordance with the selected draw tool and draw tool options. With the airbrush 
example, the system displays the curved line repeating itself with the chosen color. 
Pressing the side bar while repeating the motion will allow selective removal of any 
edits to the image using a curve established between the anchor points. 

Detailed Deacripti nn Text (53) : 

If the side button has not been pressed, a test is made at block 126 to determine if 
the tip of the pen has been pressed. If the tip has not been pressed, the routine 
loops to block 124. Otherwise, a test is made at block 128 to determine if the tip 
was released prior to moving the pen. If the pen has been moved while the pen tip 
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was pressed against the tablet, the imaging program enters freehand draw mode, shown 
at block 130. In one embodiment of the invention, freehand draw mode is entered if 
the pen moves the equivalent of three or more pixels. While in freehand draw mode, 
freehand edits may be made to the -imagf* in a manner similar to prior art imaging 
programs. This mode will remain xintil pressure on the pen tip is released. After 
beginning freehand draw mode, the routine loops to block 124. 

^p^a^led Description Text (58) : 

To further exemplify the advantages of the CD draw tool described in FIG. 5, an 
original (unedited) _imag:e 130 that is representative of an image displayed on a 
monitor is illustrated in FIG. 6. A main menu 132 is displayed across the top of the 
image to allow a user to select editing, viewing and printing options, as discussed 
in FIG. 3 and accompanying text. The main menu 132 is from an embodiment of the 
aesthetic imaging system 20 . 

Detailed Description Text (59) : 

FIGS. 7A-7E and 8A-8E contrast exemplary steps taken to make identical edits to the 
•image 130. The steps required to modify the image using a relatively advanced prior 
art imaging system are illustrated in FIGS. 7A-7E. These steps are modeled after a 
prior art imaging system that has been distributed by Mirror Image Technology, Inc., 
a division of Virtual Eyes, Incorporated, the assignee of the present invention. The 
steps required using an embodiment of the aesthetic imaging system 20 in accordance 
with the invention are illustrated in FIGS. 8A-8E. Briefly, each set of drawings 
illustrates examples of edits to a patient's nose, chin, and neck regions. The edits 
are for use in explaining the invention only, and merely exemplary in nature. 

Deha-ileH De flrri pi- i r>n Text (143) : 

FIG. 9A illustrates a user interface for a contour tool for use in editing images in 
accordance with the invention. The contour tool is invoked from the main menu of the 
imaging program, as indicated at block 200. The contour tool has similarities to a 
blend tool, but utilizes pixel manipulation to pull pixels from one area to another. 
For example, the tool works great for chin and lip augmentations. 

Detailed Description Text (147) : 

FIGS. 9B-9C further describe the operation of the contour tool, by illustrating how 
pixels are replicated from one area of an image to another. The image areas 
described are for exemplary purposes only, and are simplified for clarity in this 
discussion. With reference to FIG. 9B, an area 209 of an image is comprised of red 
218R, green 218G, blue 218B, and yellow 218Y areas separated by boundary lines 210, 
212, 214, and 216. It is assumed that a pair of anchor points 218 and 219 have been 
established by a user along the boxindary 212, wherein the aesthetic imaging system 
will display a line segment 220 between the two anchor points. 

^>e^ane d Desrr-i pt i nn Text (150) : 

If the opacity level is less than 100 percent, pixels from the underlying image 
areas that are being written over by the newly defined areas will be blended into 
the newly defined areas. At an opacity of 80 percent, for example, the area bounded 
by segments 220 and 221 will still be primarily blue, but the portion of this area 
bounded by the segment 220 and the boundary line 214 may have a yellow tinge; and 
the portion of this area bounded by the boundary line 212 and segment 221 may have a 
green tinge. As the opacity percentage is dropped, the effects on these areas will 
be even greater. 

Detailed Degrription Text (153) : 

FIG. 9D is a "bef ore"_picture without any modifications. To edit the right half of 
the patient's upper lip, the user first selects the desired end points surrounding 
the feature to be modified. A first anchor point 374 is designated near the middle 
of the upper lip, and a second anchor point 376 is designated at the right outer 
margin of the upper lip. When the anchor points have been selected, a line 3 78 is 
displayed between the points by the aesthetic imaging system. By floating the pen 
over the tablet in a direction generally indicated by arrow 380, the line is bent to 
form a curve 382 that approximates the contour of the feature being edited. When the 
user has fitted the curve to the feature, the shape of the curve is set by tipping 
the pen. As shown in FIG. 9E, the user then selects a point along the portion of the 
curve by tipping the pen at the desired location. By floating the pen above the 
tablet, the user can stretch the selected feature in the manner described above. In 
FIG. 9E, the user floats the pen in a direction generally indicated by arrow 384 to 
enlarge the patient's right upper lip. As the upper lip is expanded, the pixels 
foming the upper lip are replicated to expand the portion of the lip, while the 
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pixels outside of the upper lip are deleted. 



Dfifailpd npgrr-ip^if^n Tpvt (158) : 

The black dots are brush size options that allow a user to choose the thickness of a 
shaping tool. The zoom option allows a user to look at an image in greater detail. 
When the zoom option is invoked, the aesthetic imaging system displays a square 
overlaid on the image. The square can be positioned by the user with the pen. After 
positioning the square, that area of the image will be magnified when the pen is 
selected. Canceling with the pen will display normal viewing mode. The undo option 
allows a user to undo edits to an image . The compare option allows a user to 
transition between before and after images. The split image option allows a user 
view before and after images side by side. The inverse option creates the mirror 
image of all or only a portion of an -image that has been designated by the user. The 
blend tool will blend edits with the surrounding area. Many of the options shown are 
also implemented as separate tools under View in the main menu. These are described 
in greater detail below. 

Dej-a-iled Deacription Text (160) : 

At block 232, the user is prompted to designate an area of the image to be edited. 
In a preferred embodiment, this is accomplished by pressing down on the pen and 
freehand drawing an area, e.g., a circle, that is to be subject to the edit. In this 
regard, the imaging program contains a unique feature wherein if a partial area is 
designated and the pen subsequently released, the drawing area will automatically be 
formed into a contiguous area by the imaging program. At block 236, a test is made 
to determine if an area has been designated by the user. If not, the routine loops 
back to block 234 and awaits a designation. 

neta-iled DeHrr-ipt-i on Text (161) : 

After an area has been designated, any edits to the designated area of the image are 
performed in accordance with the selected shape tool, as indicated at block 236. Two 
exemplary shape tools for editing an image are illustrated in FIGS. 12A and 12B. At 
block 238, a test is made to determine if editing of the designated area is 
complete. In one embodiment, this involves testing for when the user "selects" with 
the pen anywhere on the tablet. The routine remains at block 238 until editing is 
complete (or the user exits using the side button) . Upon completion of the edits, an 
autoblend box is displayed in the vicinity of the edited area, as indicated at block 
240. 

Dfttailftd Description Text (163) : 

FIG. 11 illustrates an example of an autoblend box 250 that may be drawn by the 
aesthetic imaging system in accordance with the invention. As is discussed above, 
the autoblend box 250 may be used to: (1) move an edited area, (2) paste the edited 
area onto the image while blending the edge created between the edited area and the 
rest of the i mage , and (3) paste the edited image without blending. While the 
autoblend box 250 uses the conventions set forth below, those skilled in the art 
will appreciate that other conventions may be used without departing from the scope 
of the invention. 

Detailed Degnrip^inn Text (165) : 

With reference again to FIG. 10, a test is made at block 244 to determine if the pen 
tip has been pressed at a location outside of the autoblend box. If the pen tip was 
pressed at a location outside the autoblend box, the edit area is pasted with a 
blending of the edges at block 260. Otherwise, a test is made at block 262 to 
determine if the tip location was within the approximate three-, six-, nine-, and 
twelve -o ' clock areas of the autoblend box. A pressing of the tip within any of these 
areas results in the edited area being pasted without blending, as indicated at 
block 264. As indicated at block 266, a selection in a location in the autoblend box 
apart from the three-, six-, nine-, and twelve -o * clock areas will allow the image to 
be moved. In this case, the edited area will track movement of the pen as long as 
the tip remains pressed. After a move is completed, the routine loops to block 242. 

Detailed Description Text (166) : 

If a paste has been accomplished using blocks 260 or 264, a test is made at block 
268 to determine if the user wishes to exit the shape routine, e.g., by pressing the 
side button. If not, the routine loops to block 232 where a new area of the image 
may be considered. Otherwise, the routine returns to block 82 of- FIG. 3. 

Detailed Dear ript inn Text (167) : 

FIGS. 12A and 12B illustrate two exemplary shape tools that are available when using 
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the aesthetic imaging system. With reference to FIG. 12A, a cutout tool is unique in 
that a user can select an area of the imagf^ to be cut out, thereby creating a "hole" 
in the—image^ and an identical -imagfi underneath the cutout image caji then be moved 
in all directions as it is viewed through the hole. The cutout feature is especially 
useful for profile views including chin augmentation, brow lifts, and maxillary and 
mandibular movement; and frontal views, including otoplasty, brow lift, lip 
augmentation, nasal base narrowing, and maxillary and mandibular movement. 

np^ailf^d Dfiffcription Text (168) : 

At block 2 70, the current^Lmage is copied to a working buffer. As is discussed in 
FIG. 10, when the cutout subroutine is called the user has defined an area of the 
•imagp to be edited. At block 272, a boundary is created around the designated area 
designated in block 234 of FIG. 10. At block 274, the working area is displayed 
inside the boundary, and the current image; displayed outside the boundary. In this 
manner, the imagf> in the working buffer can be moved relative to the imagft in the 
current buffer until the desired alignment has been achieved. The program then 
returns to the routine of FIG. 10. 

npt-a-ilfid Df^jflr-ription Text (170) : 

With reference to FIG. 12B, a rotate tool in accordance with the invention is 
particularly useful when editing profile views including the nasal tip, mandible, 
maxilla and brow areas; and frontal views including nares, brows, and the comers of 
the mouth. As is discussed in FIG. 10, when the rotate routine is called, the user 
has defined an area of the imagg* to be edited. At block 276, the area designated in 
block 234 of FIG. 10 is shown in phantom. At block 278, the imaging program waits 
for the user to enter an axis of rotation. An axis is then entered by the user, as 
indicated at block 279. 

npl-ailp^d npfir-r-i pl-inn Tpy^ (171) : 

Once an axis of rotation is entered, a display line emanating from the axis point is 
displayed on the monitor, as indicated at block 280. Also, the number of degrees of 
rotation is displayed. The position of the pen dictates the degree of rotation. As 
the pen is moved away from the axis point, the display line will lengthen, providing 
the user greater control of the rotation of the designated area. At block 282, the 
system waits for the user to enter a desired degree of rotation. The degree of 
rotation is entered by the user by selecting with the pen, as indicated at block 
283. Once the degree of rotation is entered, the designated area is redrawn onto the 
current image , as indicated at block 284. 

DRt-ailf^d Pp^ficr-iphinn Tf^xt (174) : 

FIGS. 18A-18D disclose the capcLbilities of the warp shape tool in the aesthetic 
imaging system. The warp shape tool is a powerful tool that allows users to easily 
edit a patient's features. Similar to the contour tool, the warp tool allows a user 
to select and manipulate a feature of the patient's image , with the aesthetic 
imaging system automatically redrawing the area surrounding the manipulated feature 
as the manipulation is being performed. 

Detailed DeRcriptlon Text (175) : 

As shown in FIG. 18A, the user first defines an image to be manipulated by the warp 
tool by encircling the portion of the image to be edited as shown by dotted line 
386. Once the area has been selected, a user may tip the pen to designate a stretch 
point within the selected area. As shown in FIG. 18A, a stretch point 388 has been 
designated near the top portion of 

Derailed nefirri pt ,i nn Text (176) : 

the patient's right upper lip. Once the stretch point has been designated, the user 
may float the pen in the desired direction that they would like to stretch the 
image . The portion of the image that is located at the stretch point is pulled in 
the direction that the user floats the pen, with the area surrounding the stretch 
point either expanded or compressed. That is, the area in the direction that the 
user floats the pen is compressed, and the area away from the direction that the 
user floats the pen is expanded. The amount of expeuision or compression is dictated 
by the distance of the area away from the stretch point, in a manner discussed in 
greater detail below. Preferably, the image is manipulated in real time, so that the 
user is presented with a seamless cuid continuous stretching or movement of the 
selected feature. 

Detailed Description Text (180) : 

when the desired warping effect is achieved, the user tips the pen to fix the 
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stretch point in a desired location. To ensure there is no discontinuity between the 
manipulation area and the surrounding area of the patient's imagp, automatic 
blending around the outer margins of the manipulation area is performed by the 
aesthetic imaging system. As shown in FIG. 19E, the aesthetic imaging system also 
remaps the warping area, creating four new rectangles 436a, 436b, 436c, and 436d 
based on the ending location 438 of the stretch point. In a preferred embodiment of 
the invention, a second stretch point may then be selected within the manipulation 
area and the process repeated, with the second warping transforming rectangles 436a, 
436b, 436c, and 436d that resulted from the first warp. After the second warping is 
completed, the user may then select a third or additional warp point to further 
manipulate the image. Each manipulation is performed without the user having to 
redefine the manipulation area. 

np>^a^^f>^ ^ nf>.q^!^r^p^^r^n Tf^vi- (182) : 

The warp tool with multiple stretch points is a very powerful tool as it allows the 
user to quickly manipulate an image with a minimum use of drawing tools or piecemeal 
editing. Because the warp tool performs the manipulation in real-time, the edits are 
accomplished very quickly and fluidly. A user may therefore generate a desired image 
in a minimal amount of time. 

npt-a-i1f><i Dpfi^!•r ^p^^^)n Tpvt- (183) : 

Viewing an Image 

Dfit-a-iled Description Text (184) : 

FIG. 13 illustrates an exemplary routine for implementing the view features of the 
imaging program. In FIG. 13, the solid blocks indicate user interface options 
presented to the user by the aesthetic imaging system and the dashed blocks 
represent system responses to the decisions made. The view group includes: Compare, 
Prioritize, Split image , Mirror Image, and Restore to Original, as well as other 
options including Zoom and Emboss. At block 300, a test is made to determine if the 
Compare option has been selected. 

Deta-ileH Deffnr-i pf i r>n Tevt. (185) : 

The Compare option allows a modified image to be compared to the original image so 
that a viewer can more readily see the chamges. Specifically, as the pen is floated 
from the top to the bottom of the tablet, the user will see one image transition or 
"morph" into the other. The moyphing is accomplished by overlaying the original 
•image with the modified image, and varying the opacity of the modified image . When 
the modified image is opaque, only the modified image may be viewed by a user. When 
the modified image is completely transparent, only the original image may be viewed 
by a user. In between these two extremes, varying amounts of the edits made to the 
image will be apparent to the user. The feedback to the patient as the original 
i mage morphs into the modified image is much more powerful than a side -by- side 
comparison of the two images - 

Detailed Description Text (186) : 

When a desired comparison level is achieved, a user can press the tip of the pen to 
freeze an image displayed on the monitor at a point anywhere from zero to 100% of 
the transition from the original to the modified image . Freezing an image at a 
partial transition is extremely helpful where edits have been performed on an image 
that are not realistically achievable in surgery, but an achievable result lies 
somewhere between the original and the modified image . For example, it is easy to 
edit a blemish on a face so that area resembles the surrounding skin and thus 
becomes invisible. However, the total removal of the blemish may not be realistic. 
In this case, a transition of that area toward the original image will slowly "fade 
in" the blemish. A physician may then freeze the fading process at a desired point 
to provide a realistic image of what surgery can achieve to the patient. 

Derailed Desn-r i phi on Texh (187) : 

Another option that is a subset of the Compare option is the Prioritize option. The 
Prioritize option allows a user to designate areas that have been edited so that the 
user can selectively illustrate the effects of two or more procedures that have been 
shown to a patient. For example, FIG. 14A illustrates a modified profile image 302 
of a patient that includes a rhinoplasty procedure (nose) 304, a chin augmentation 
procedure 306 and a submental lipectomy procedure (neck) 308. The boundaries that 
have been edited are illustrated by dashed lines 304a, 306a, and 308a, corresponding 
to the patients original nose profile, chin, and neck, respectively. Using the 
Prioritize option, the user can designate one area on the modified image , and 
illustrate transitions between the original and modified images at that area only by 
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floating the pen. Any areas not selected will continue to be displayed as the 
original image. 

Df^ta-ilpH npRr?r-iph-inr> Tf^Yh (188) : 

With reference again to FIG. 13, if the Compare option has been selected, a test is 
made at block 309 to determine if the entire -imagf^ is to be compared or only certain 
portions of the_Jjnage^ i.e., using the Prioritize option. If less than the entire 
image* is to be compared, the user is prompted to enter the area or areas that are to 
be compared at block 310. A user may then define one or more "priority areas" by 
freehand circling the desired area. When the priority areas or defined, or if all of 
the edits are to be reviewed during the comparison, at block 311 the user is 
prompted to float the pen in a vertical motion on top of the tablet to transition 
between the original and modified images, in accordance with the Compare feature 
discussed above. An illustration will clarify this point. 

np^a^^pr^ DPRr-riph-i on Tf^xt (189) : 

In FIG. 14B, a first priority area 312 has been defined that corresponds generally 
to the nose. Given this selection, the nose area only will transition from original 
to modified as the pen is moved, with the rest of the image being displayed 
unedited. Thus, the modifications to the chin and neck no longer are shown. In FIG. 
14C, a second priority area 314 has been defined that corresponds generally to the 
chin. The first priority area 312 has been kept. Given these selections, the nose 
and chin areas only will transition from original to modified as the pen is moved, 
with the rest of the imagg^ being displayed unedited. Thus, the modifications to the 
neck are not illustrated. 

Dc^ha-nf^r i npfir-r-i phi nn Tf>yr. (190) : 

With reference to FIG. 14D, a third priority area 316 has been defined that 
corresponds generally to the neck, along with the former designations. Given these 
selections, all three priority areas 312, 314, and 316 transition with movement of 
the pen. Again, the undesignated portions of the image are displayed in an unedited 
form, even if parts of the image outside the priority areas have been edited (no 
edits are shown). Because edits have not been made, from a user's standpoint the 
transition in FIG. 14D appears to be a comparison between the original and modified 
Images. 

Detailed Description Text (191) : 

While preferably the user defines the priority areas on the image being edited, it 
will be appreciated that the priority areas may also be automatically defined by the 
aesthetic imaging system. A comparison may be made between an original image stored 
in a buffer and the edited image that has been modified by the user. Any areas 
containing differences between the original and the edited image may be highlighted 
by the aesthetic imaging system, and a priority area automatically defined for each 
area containing differences. Whether a priority area is defined may also depend on 
the number of differences between the original image and the edited image . 

net-ailed DeRnri p^ i r>n Tevh (193) : 

In the alternative embodiment in which textual identifiers are assigned to each of 
the priority areas, it will be appreciated that various methods can be used to 
display to the user the correspondence between the textual identifier and the 
priority area. For example, when the user selects one of the textual identifiers, 
the priority area could be highlighted or otherwise encircled with a contrasting 
color to indicate to the user the area of the image that corresponds to that textual 
identifier. Similarly, a user may also point to specific areas of the displayed 
image and have the textual identifier corresponding to that area appear to the user. 
Displaying the correspondence between priority area and identifier would allow a 
user to rapidly determine the available priority areas that may be shown to a 
patient when the user has not recently worked with or otherwise viewed the image. 

Detailed Description Text (194) : 

with reference again to FIG. 13, at block 320 a test is made to determine if the 
user wishes to save a transitional or morphed view of an image . If a transitional 
view is to be saved, the user may establish the percentage transition, i.e., 
anywhere from zero to 100 percent transition (zero percent being the original image 
and 100 percent being the edited image) , by floating the pen up or down above the 
tablet to establish the view, and the pressing the tip of the pen against the tablet 
to freeze the transitional image, as indicated in block 322. If the tip is pressed 
again, the frozen image is saved. The save options are available with or without the 
priority areas in effect. After the save has been accomplished, or if the user did 
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not wish to save a transitional view of 
the routine branches to block 326. 



the Compare option is complete and 



nfihailfir i DRfin-r-iphi nn TP>Y^ (195) : 

In an alternative embodiment of the invention, the Conqpare option allows a user to 
compare a modified imagf> with any edits made to the modified image during the 
current editing session, i.e., before the changes are permanently saved to the 
modified imags. Specifically, with reference to FIG. 2, this embodiment of the 
Compare option contrasts the imagf^ in the current image buffer 54 with the image in 
the modified -imagfi buffer 52. As discussed above, this embodiment of the Compare 
option may also be used in conjunction with the Prioritize option to allow the user 
to select priority areas for comparison. In this case, the priority areas transition 
from the modified to the current image , while the modified Image only is displayed 
in the other (nonselected areas) areas. 

Dpfa-ilF^d nf^H^!rip^,ion Texh (196) : 

At block 326, a test is made to determine whether the user wishes to view a split 
image. The Split image option is used on a frontal picture only, and allows a 
patient to see his or her asymmetries. If a split image view is desired, the user is 
prompted to select an image , e.g., original or modified, at block 330. At block 332, 
a vertical centerline is displayed on top of the selected image. The user is then 
prompted to position the centerline at the location desired, as indicated at block 
334. Typically, the centerline will be positioned to dissect the image into equal 
halves, using the nose and the eyes as reference points. At block 336, the aesthetic 
imaging system displays two -imagepi^ one showing the left halves pieced together and 
the other the right halves pieced together. Specifically, the aesthetic imaging 
system will produce an inverse -jmage of the left (right) half and then add it to the 
left (right) half 

neha-iled Dear ri phi r>n Tey^, (197) : 

FIGS. 15A-15C illustrate the resultant imagpFi that are displayed when the Split 
Tmage option is invoked. In FIG. 15A, a frontal image 350 of a patient is shown, 
including a centerline 352 that has been positioned at the center of the patient by 
a user. FIG. 15B is an illustration of the left halves of the image after being 
pieced together by the aesthetic imaging system, as indicated by reference numeral 
354. FIG. 15C is an illustration of the right halves of the image, as indicated by 
reference numeral 356. With the Split Image option, patients can view what they 
would look like if their faces were symmetrical. The tool is especially useful in 
the consultation stage because many people do not realize that the typical face is 
asymmetrical, and changing a face to be perfectly symmetrical, if possible, is not 
necessarily desirable. 

Detail er i ne.gnT-i pt i nn Text (198): 

Once the split images have been displayed, or if the Split image option was not 
selected, a test is made at block 360 to determine if the user wishes to view an 
inverse or mirror image of a pi^!^^ure, e.g., to show patients the view they see of 
themselves when looking into a mirror. If an inverse image is desired, the user is 
prompted to select an image to be viewed at block 362. The selected image is then 
"flipped" using the aesthetic imaging system, and displayed on the monitor, as shown 
at block 364. After the image has been displayed, or if the inverse image option was 
not invoked, routine returns to block 86 of FIG. 3. 

De^aileH near-ri pi- i nn Tey^ (199) : 

When viewing an image on the monitor, it is important that a patient remains focused 
on the image being manipulated rather than on the features of .the aesthetic imaging 
system. When certain viewing options are selected, for example, the prioritize 
option shown in FIGS. 14A-14D, or the split image option shown in FIGS. 15A-15C, the 
menu bar normally located across the top of the display is therefore removed. Only 
the image of the patient is kept on the screen, ensuring that the patient remains 
focused on the image being manipulated. 

ne^ai^eH Deffrriptinn Text (200) : 

In certain situations where it is necessary to display a menu bar, however, it is 
advantageous to minimize the visual appearance of the menu bar. FIG. 20 is a 
representative image 410 of a patient with a menu bar 412 located across the top of 
the screen 414. To reduce the distraction caused by the menu bar, the menu bar is 
preferably translucent to allow the user to view the patient's image through the 
menu bar. The text 416 and the line 418 indicative of the menu bar are preferably 
presented in a contrasting, yet muted, color to allow the user to read the commands. 
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For example, the text and the line outlining the menu bar may be presented in an 
off-white. While editing the -imagp with a patient present, the patient is therefore 
not overly distracted when the menu bar periodically appears at the top of the 
screen. Even when the menu bar is present for extended periods of time, a patient is 
not distracted since it does not visually stand-out from the patient's image . At the 
same time, a skilled user is provided with sufficient information about the menu 
choices to allow them to choose the appropriate menu options. 

nf*ha-i1pri Df^ffrr-ipt-inn Tf>yt (202): 

FIG. 21 is a flow chart of an exemplary routine for implementing a zoom feature in 
the aesthetic imaging program. The zoom feature allows a user to increase the scale 
of the image to better view a selected area and to improve the aibility of the user 
to edit fine details in the image. At a block 400, a zoom point is selected by a 
user- The zoom point identifies the center of the image to be expanded under the 
control of the user. At a block 402, the pirhnriq is adjusted to position the zoom 
point at the center of the monitor. Centering the picture ensures that as the image 
is enlarged, the portion of the image surrounding the zoom point will be displayed. 
At a block 404, the user is allowed to input a desired magnification factor. 
Preferably, the magnification factor is selected by floating the pen from the bottom 
(minimum magnification) to the top (maximum magnification) of the tablet. As the 
user floats the pen over the tablet, the image on the monitor is correspondingly 
magnified and redisplayed at a block 406. With each redisplay of the image, at a 
block 408 a test is made to see if the user has frozen the image by pressing or 
tipping the pen. Once the -image is frozen at a desired magnification, a user can 
manipulate the -iniage using the array of drawing tools described above. It will be 
appreciated that for very fine work, such as removing small wrinkles surrounding a 
patient's eyes, the ability to magnify the image increases the quality of the 
editing that may be achieved. 

net-ail efi Defirr-jptinn Teyt (203) : 

Several refinements of the zoom feature may be incorporated in the aesthetic imaging 
program to improve the results of the zoom. For example, a smoothing function may be 
incorporated in the zoom feature to ensure that as the image is magnified it does 
not become "pixelly" or grainy. The smoothing function may be implemented in 
software. Preferably, however, the smoothing function is implemented in hardware, 
such as a smoothing feature provided in the Targa 2000 board described above and 
incorporated in the aesthetic imaging system. Further, feedback may be provided to 
the user in the form of a numerical display on the image to indicate the approximate 
magnification as the user floats the pen from the bottom to the top of the tablet. 
Other means can also be implemented to allow the user to select the desired 
magnification, including a pull-down menu or numerical entry. 

Deha-ile fi Defirriptinn Tey^, (204) : 

Yet another view option provided to the user in the aesthetic imaging system is an 
Emboss option. It has been fo\ind that the emboss viewing option is very helpful in 
allowing a user to discern wrinkles or other skin imperfections in a displayed 
image. By selecting the emboss option, the user causes an image of the patient to be 
displayed in a gray scale. The emboss option displays an image that is similar to an 
etching made of a three-dimensional raised surface. A two dimensional image is 
portrayed, with the depth of the raised surface indicated by a darker shade of gray. 
The emboss option removes any deceptive information conveyed by the color or shading 
of the skin of the patient and allows any raised or depressed areas to be clearly 
highlighted. When viewing wrinkles or other imperfections on a patient, the emboss 
option therefore clearly identifies the raised or depressed features over the smooth 
skin of the patient . 

ne1-ai1ed DeRr-ription Text (207) : 

dx and dy =the nf f F!e^. of the emboss operation; and 
Detaileri DeRrript-.i on Text (209) : 

After selecting the emboss option, the user may manipulate the offset of the emboss 
by floating the pen over the tablet- Floating the pen parallel to the x-axis of the 
tablet adjusts the dx parameter in the emboss function, and floating the pen 
parallel to the y-axis adjusts the dy parameter in the emboss function. It has been 
found that various features on the skin of a patient can be brought into greater 
clarity by adjusting the of f RetR of the embossed i mage . A user may therefore focus 
on the desired feature that they wish to show a patient. When the desired of f iqehR 
are selected, the user may freeze the of f RetR by tipping the pen on the tablet. A 
user may then adjust the scale parameter by floating the pen parallel to the x-axis 
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of the tciblet. Again, it has been foiind that various skin features can be brought 
into greater clarity by adjusting the pressure of the emboss. When the desired 
pressure results are achieved, the user may freeze the imagp^ by tipping the pen on 
the tablet. 

npta-ilf^ri Dp>F;rri p^•ir>n Paragraph Tahiti (1) : 

Copy, Cutout, Resize Stretch, Rotate Freeze 

Compare Zoom Zoom Undo Undo Compare Compare 

Split imagp Split -imagf^ Split Image Inverse Inverse Blend Blend 



CLAIMS : 

1. A user interface for an aesthetic imaging system containing digitized images of a 
patient, the aesthetic imaging system allowing a user to manipulate a patient i mage 
to display potential results achievable through cosmetic surgery, the aesthetic 
imaging system including a processor, a memory, a monitor, and means for controlling 
a cursor displayed on the monitor, the user interface comprising: 

(a) a display area for displaying a digital image of a patient comprised of a 
plurality of color pixels; and 

(b) a command menu area for displaying a plurality of commands that may be used to 
manipulate the digital -jmage of the patient, the command menu area substantially 
overlapping the display area so that a portion of the digital image is positioned 
behind the command menu area, the command menu area being substantially transparent 
to allow the portion of the digital -imagp^ positioned behind the command menu area to 
be visually apparent to a user, the plurality of commands in the command menu being 
substantially opaque and visually distinguishable from the portion of the digital 
image positioned behind the command menu area to allow the user to select from the 
plurality of commands. 

3. The user interface of claim 2, wherein the plurality of commands in the command 
menu area are removed during editing of the digital image. 
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TITLE: Method and apparatus for positioning an instrument relative to a patients 
body during a medical procedure 



Abstract Text (1) : 

A method and apparatus for annotation of medical imagery to facilitate patient 
identification, diagnosis, and treatment is characterized by an imaging device for 
producing a first signal representative of sensed characteristics of the individual 
and a minutiae generator which receives the first signal and produces a second 
signal representative of minutiae of the individual, the minutiae corresponding to 
specific branch points of blood vessels of the individual. A minutiae data generator 
analyzes the characteristics of minutiae and produces a third signal representative 
of the characteristics which is stored in a minutiae database for each of the 
plurality of known individuals and their medical conditions. The minutiae and 
minutiae data may be used to annotate medical imagery to facilitate subsequent imagft 
comparison by providing standardized registration points and time-varying 
characteristics. A minutiae matcher pairs corresponding second signals and third 
signals from a current patient with those from a database record, and the paired 
signals are used to align the_imagea and compare them. The minutiae analysis 
techniques of the invention can be used to identify medical patients, assist in the 
diagnosis of medical conditions, and detect and monitor the use of alcohol and 
drugs, including anesthesia. 

RT-if>f fin mma-ry Tp>v^■ (2) : 

This invention relates generally to the field of imagft recognition and processing 
and specifically to methods and systems for identifying, diagnosing, and treating 
people based on thermal minutiae within a person's body, primarily the face. 

Brief Summary Text (10) : 

Inkless techniques are now generally used to produce a "tenprint" card which 
siibstitutes for the former rolled print card. Common inkless techniques utilize 
polarized light to illuminate the fingers, and light sensors to image the light 
reflected and refracted from the ridges. The resulting image can be more consistent 
and higher quality than the rolled prints, since inconsistencies in the amount of 
ink applied and in the pressure used to transfer the print to paper are not a 
factor. 

R-ripf Nummary T^yt (19) : 

A known algorithm of the National Institute for Standard Technology can be used to 
compare a previously stored electronic imagf^ of minutiae coordinate locations with 
the minutiae locations identified and stored by the computer. 

Brief Summary Text (20) : 

U.S. Pat. No. 5,040,224 to Kara discloses a fingerprint processing system capable of 
detecting a core of a fingerprint image* by statistically processing parameters. 
Harass invention provides a system to determine a core in the fingerprint image 
and/or to detect directions and curvatures of ridges of the fingerprint image prior 
to detection of the position of the core. This reference defines minutiae as abrupt 
endings, bifurcations, and branches. 

Bric^f Summary Text (21) : 

U.S. Pat. No. 4,790,564 to Larcher teaches a process and apparatus for matching 
fingerprints based upon comparing the minutiae of each print in a database with 
precomputed vector images of search minutiae in a search print to be identified, 
comparing position and angle, a result of such comparison being a matching score 
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indicating the probability of a match between the angle of a file print minutiae and 
the angle of precomputed vector images of the search minutiae. Over an under- inking 
of a rolled print can change the apparent type of minutiae associated with a 
particular point from one printing to the next. However, not all corresponding 
minutiae will appear to change type in the two pairs. Therefore, matching for type 
as well as for x and y coordinates provides a stricter match requirement and results 
in better system accuracy. Larcher assigns higher values to minutiae which match in 
x,y and type. 

RT-ipf finmnary Tf>yt (22) : 

As Larcher points out, there are advantages to matching minutiae rather than the 
entire i magfq of the fingerprint in itself. An elementary matching operation 
comprises the comparison of two sets of minutiae, i.e., two sets of points, each 
point having three coordinates x, y, and a. An elementary matcher attempts to 
superimpose the two sets of points, in order to count the number of minutiae which 
are common to the two fingerprints. 

U-rHf>f fliiTTima-ry Tpy^ (27) : 

Other known approaches compare two sets of image features points to determine if 
they are from two similar objects as disclosed for example in Sclaroff and Pent land, 
MIT Media Laboratory, Perceptual Computing Technical Report #304. This reference 
suggests that first a body- centered coordinate frame be determined for each object, 
and then an attempt be made to match up the feature points. 

Brief Summary Text (31) : 

Second, the image is divided into pixels, where the size of the pixel relates to the 
quality of the result desired; 

Rriftf Summary Text (33) : 

Fourth, each minutia is assigned a vector having magnitude and directional 
information in relation to the surrounding characteristics of the fingerprint. 
Typically for each fingerprint, there would be a siibstantial number of minutia 
vectors characterizing its image; 

R-r-ief RTimmary Tevh (39) : 

Still another known scheme utilizes eigenanalysis of visual face i mages to develop a 
set of characteristic features. Pentland, View-Based and Modular Eigenspaces for 
Face Recognition, MIT Media Laboratory Perceptual Computing Section, Technical 
Report No. 245. Faces are then described in terms of weighting on those features. 
The approach claims to accommodate head position changes and the wearing of glasses, 
as well as changes in facial expressions. This disclosure teaches that 
pre-processing for registration is essential to eigenvector recognition systems. The 
processing required to establish the eigenvector set is extensive, especially for 
large databases. Addition of new faces to the database requires the re-running of 
the eigenanalysis. Accordingly, use of eigenanalysis may not be appropriate for use 
in a general face identification system such as would be analogous to the FBI's and 
AFIS fingerprint system. 

Brief Snmrnary Text (42) : 

One known scheme using facial thermograms for identification is described in the 
Prokoski et al U.S. Pat. No. 5,163,094 which discloses defining "elemental shapes" 
in the surface thermal image produced by the underlying vascular structure of blood 
vessels beneath the skin. Depending on the environment of use, thermal facial 
identification may provide greater security over identification from visual images 
and may therefore be considered preferable. It is extremely difficult, if not 
impossible, to counterfeit or forge one face to look like another in infrared, 
whereas it is often possible to disguise one person to look like another in visible 
light. However, the use of elemental shapes is found in practice to be vulnerable to 
such variables as head rotation and tilt, ambient and physiological temperature 
changes, variations in imaging and processing systems, and distortions or 
obstructions in a facial image (e.g., due to eyeglasses). 

Brifsf Summary Tfixt (43) : 

Eigenanalysis of the elemental shapes of a thermal facial image has also been used 
for recognition. In one approach, several sets of elemental shapes are produced for 
each image by imposing different thermal banding constraints. The totality of shapes 
are then analyzed with respect to a library of facial thermal Imagfts. Eigenshape 
analysis is used to compare the characteristics of shapes in each person' s images . 
Eleven characteristics of each shape are considered, including: perimeter, area. 
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centroid x and y locations, minimum and maximum chord length through the centroid, 
standard deviation of that length, minimum and maximum chord length between 
perimeter points, standard deviation of that length, and area/perimeter. 

Rr-if^f flnmmary Tf^yh (44) : 

Each person's image is then characterized by set of 11-coef f icient vectors. The 
difference in eigenspace between any two i magf^g is calculated to yield a measurement 
to which a threshold was applied to make a "match/no match" decision. In practice, 
such a system yields a useful method and apparatus for some applications. However, 
the calculation techniques for such a system are computationally intensive and 
require additional computational analysis of the entire database when new amflgpifi are 
added. As with others of the prior known techniques, recognition is seriously- 
impacted by edge effects due to head rotation and tilt, and by loss of definition in 
very cold or very hot faces. 

Brief Summary Text (46) : 

Therefore, the need remains for a system and method that can be used to reliably 
recognize and verify the identity of an imaged person without manual assistance and 
without cooperation from the person being identified. 

Drawing Depirriph-i on Text (11) : 

FIG. 8 is a block diagram of the apparatus for annotating an image of the human body 
according to the invention; 

n-raw-ing DeRr'-ri pt- i r>n Tevh (12) : 

FIG. 9 is a block diagram of a modified apparatus of FIG. 8 for identifying 
reference minutiae in an annotated^Jjnage; 

Drawing Description Text (13) : 

FIGS. 10a, 10b, and 10c illustrate facial minutiae superimposed on three different 
thermal images, respectively, of the same face; 

Drawing Description Tey^ (14) : 

FIG. 11 is a block diagram of an apparatus for annotating a medical image of the 
human body according to the invention; 

Detailed Description Text (4) : 

In operation, thermal imaging device 102 obtains a thermal image of the face of 
person 101. A digital signal representative of the thermal image is provided as 
input to minutiae generator 104, which generates signals representative of thermal 
facial minutiae points for 101. These minutiae points have a number of 
characteristics including a specific location within the person or relative to other 
minutia, the apparent temperature at a given time, the temperature signature over a 
period of time, whether the minutia corresponds to a vein or artery, the width of 
the blood vessel, and the vector direction of branching blood vessels from the 
minutia. These and other characteristics are sensed and data relative thereto are 
generated by a minutia data generator 105. This data is stored in the minutiae 
database 111. The minutiae matcher 106 compares minutiae data for known individuals 
which has been stored in the database with that for unknown individuals currently 
being imaged by the imaging device 102. If a match is detected, a corresponding 
signal is sent to the output display/processing device 114. 

Pet-ailed Dearri pi- 1 nn Tevt (7) : 

Thermal imaging device 102 may be any device that produces a signal representative 
of the thermal characteristics of the face portion 101. In a preferred embodiment, a 
conventional digital video camera sensitive to thermal energy is used for the 
thermal imaging device 102. As described herein, it is found that tractable imagery 
for facial identification may be derived from passively obtained infrared images of 
facial heat emanations which can be detected by commercially available thermal 
imaging devices sensitive in the 3 to 12 micron wavelength band. Unlike fingerprints 
that are characterized by a limited range of intensity values corresponding to three 
dimensional ridges which are essentially concentric rings about a single center, 
plus anomalous arches, line endings, and bifurcations, facial thermograms are 
generally characterized by continuously varying wide distribution of temperatures, 
including multiple maxima and minima values. Where the skin surface is unbroken, 
there is gradual variation of temperatures from the hot areas on either side of the 
nose to the relatively cool areas of the ears and checks. The eyes appear to be 
cooler than the rest of the face. The nostrils and mouth, and surrounding areas, 
will look warm or cool depending upon whether the subject is inhaling or exhaling 
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through them. Discontinuities in the skin surface temperature may be evident where 
scars, moles, bums, and areas of infection are foiind. 

Df^t-a-ilRH nf^gn-r-iptinn Tf>y^ (8) : 

In some applications, thermal imaging device 102 may be adapted for attended 
operation using cooperative persons 101 aind a human supervisor, as with 
identification systems based on rolled fingerprints. In these applications, the 
supervisor can ensure that person 101 is properly positioned, and can adjust gain, 
focus, and other parameter of thermal imaging device 102 to optimize the quality of 
the thermal— image produced by thermal imaging device 102. In other applications, 
thermal imaging device 102 is adapted for unattended, stand-alone operation, for 
instance with live scans used for access control to a remote secure facility. System 
100 can further be configured based on an expectation that person 100 will be either 
cooperative (e.g., moving to a specific requested location for optimal imaging) or 
uncooperative (e.g., a mere passer-by). In environments where uncooperative persons 
are expected, identification will be facilitated by collecting the maximum possible 
amount of data, for instance by using multiple thermal imaging devices 102 and fast 
frame (i.e., sampling) rates. Additional related data, referred to herein as "ground 
truth" data, may be collected as well to provide information on factors such as 
ambient temperature, absolute size of the imaged face, or the distance of the imaged 
face from thermal imaging device 102. 

nf^ha-ilfi H Dp^Rr-riph-ion Tf^xt. (11) : 

Various perturbations, such as facial expression changes, can distort the relative 
locations of minutiae points. This is an analogous to the deformations that occur in 
fingerprints due to movement or pressure between the fingers and the print surface. 
As described below, minutiae matcher 106 allows for some variations in the position 
and characteristics of the minutiae, as well as in the subset of minutiae which are 
seen due to the field of view of thermal imaging device 102 and to possible 
obstruction of certain areas of the face in the image. 

DRt:ailg>d r)f=^Rr;-ript ion Text (18) : 

In a preferred embodiment, minutiae generator 104 locates the left canthi area and 
the right canthi area and determines the centroid for each area. The location of the 
centroid is essentially independent of the grey scale allocation of the analog 
thermal__imag£ produced by the camera 102 . The centroids are referred to as left and 
right canthi 201, 202 herein. 

np>^a^1pd np>flr-r-i p^ i r>n Tp^vh (20) : 

Other techniques may be used for location of the face center point 209 in those 
cases where the preferred use of facial symmetry and recognizable thermal features 
does not suffice. For example, other techniques may be called for with respect to 
facial images R in which an eye patch is worn, eyeglasses are not symmetrical, only a 
partial face is imaged, the lower face is covered, or the thermal pattern of the 
face is unusually distorted. The face center point 209 may in fact be outside of the 
boundaries of the facial imagp, for instance where only a partial facial imagfi is 
obtained due to the face being partially blocked by another face or some other 
object. If the person 101 is wearing glasses, the pattern of the glasses, which 
typically block the infrared emissions from the face and thereby produce an extended 
cold area with sharp thermal discontinuity, can be used to determine approximate 
face axes. Additional techniques include manual location of the face center point 
209 and preprocessing using known techniques to locate the approximate area of the 
face center point 209. As described below, the face axes may be tested for validity 
to determine whether the image requires any such special treatment. 

Detailed Dfiscription Text (22): 

Since the known techniques for identifying left and right canthi centroids 201, 202, 
left and right nostrils 203, 204, and mouth area 205 are subject to artifacts and 
other sources of error, and since some images of faces are significantly asymmetric 
or have features that are entirely missing (e.g., due to person 101 wearing an eye 
patch or having a disfigured face) , minutiae generator 104 performs checks to help 
spot instances where these points may have been incorrectly located or where unusual 
facial -jmagg^g are encountered. First, a check is made to ensure that vertical 
central line 208 and mouth line 207 intersect within mouth area 205. Next, a check 
is made to ensure that vertical central line 208 intersects a line connecting left 
and right nostrils 203, 204 at point between left nostril 203 and right nostril 204. 
If either of these conditions is not met, the face is considered to be a special 
case calling for manual intervention to determine the best approximation for face 
axes . 
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Dpl-a-ilpd DPRrr-ipl-TnTi T^yh (24) : 

In practice, it is found that preprocessing through normalization of image size 
provides advantages in later recognition. Accordingly, minutiae generator 104 uses 
the distances between left and right canthi centroids 201 and 202 and the distance 
from face center 109 to eye line 206 to compare the size of facial thermogram 20 0 
with a standard -image size. In a preferred embodiment, linear correction in the 
vertical and horizontal dimensions is used to normalize the size of facial 
thermogram 200 to match the standard, but other normalization models could be used 
as well. 

neha-iled Deacriphinn Text (26) : 

As provided by thermal imaging device 102, facial thermogram 200 consists only of an 
ordered list of thermal values corresponding to each small portion of the Imaged 
face. Minutiae generator 104 employs the following procedure to produce thermal 
contour lines for facial thermogram 200. a. For a digitized imagpi having N bits of 
resolution, or 2.sup.N bands of thermal values, determine thermal contour lines 
having a particular "current" one of the 2.sup.N values, b. Produce minutiae in 
accordance with the steps below for the contour lines of the current value, c. 
Repeat a and b above, each time using new one of the 2.sup.N values for the 
"current" value, until the desired number of minutiae have been extracted of all of 
the possible values have been processed, d. If the desired number of minutiae have 
not been extracted, repeat the process beginning with 2.sup.N-l bands of values, and 
reduce the number of bands by 1 with each iteration, skipping those that are powers 
of 2, until the desired number of minutiae have been extracted or until no further 
reduction in bands can be achieved. 

^>P^a^^ed DeRrriphinn Text, (29) : 

In a preferred embodiment, points on a thermal contour are considered minutiae if 
they form inflection points for the contoiir. However, to avoid artifacts resulting 
in too many minutiae being selected, only inflection points for curves below a 
threshold radius will be considered minutiae. Therefore, minutiae generator 104 
selects a maximum radius of curvature to be used in determining minutiae, based on 
characteristics of system 100 such as the resolution of thermal imaging device 102, 
the lens used, the quality of the recording and processing system, the desired 
number of minutiae to be extracted, the desired sensitivity and vulnerability of the 
system to minor variations in thermal t mage , the accuracy of the three dimensional 
model for registration of the face i mage , and the magnitude of systematic and random 
errors . 

Detailed Description Text (32) : 

Minutiae generator 104 selects minutiae from the facial thermogram 200 after 
preprocessing as described above by first positioning a circle of radius R on a 
thermal contour such that the contour intersects the circle, crossing it at two 
points and dividing it with equal area in each half. Next, this circle is moved 
along the contour for as far as the contour can continue to intersect the circle at 
exactly two points while maintaining an equal area on either side. If, in so moving 
the contour, a location is found where further movement would cause the contour to 
intersect the circle at only one point, the contour has ended, and the end point is 
designated as minutia point. This situation typically occurs only at the edge of a 
facial image and only rarely within the area of the face. If a location is found 
where further movement would cause the contour to intersect the circle at three or 
more points, there is an inflection point within the circle. It can be located by 
considering the slope of the contour within the circle relative to the face axes. 
The point of maximum change in slope is then designated as a minutia point. If a 
location is fovmd where further movement would cause the contour to intersect the 
circle at no points, there is a small island area within the circle. The centroid of 
that island is designated a minutia point. 

ne^a^^eH neFtrri pti on Text (33) : 

Referring now to FIG. 5, there is shown a facial image on which minutia points, e.g, 
301, have been identified on facial thermogram 200, as described above. 

Detailed Dftscriptinn Text (37) : 

As another possibility, only centroids located near the center of the face, or in 
concave areas of the face less vulnerable to artifacts due to edges, could be used. 
In other applications, minutiae may be derived using centroids of images where all 
thermal values less than a threshold are maintained, but those higher values are 
collapsed in one band. Centroids may be added to the set as the threshold is 
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reduced. In such instance, each minutia point is characterized using at least the 
(x, y, z, T) factors mentioned above, where T is the threshold. In a variation on 
this minutiae generation technique, inflection points produced from such 
thresholding rather than the centroids may be used. 

Dfqhailf^d Deac!riptir>n Tf^xt (39) : 

An additional approach is to designate undefined locations generated by compression 
and sxibsequent expansion of the image. Specifically, facial thermogram 200 is 
compressed using wavelet of fractal -based methods and then expanded again. Because 
such compression techniques are lossy in a deterministic way, a minutia set may be 
defined as the undefined locations resulting from a comparison of the original imagp> 
with the compressed-and-expanded image . This approaching provides an additional 
advantage of compressing the data used for recognition. 

nfiha-ilRfi np>RP-ripti nn Tf^xt (41) : 

As mentioned above, it may be desirable that all thermal images be scaled to a 
standard size prior to processing. It also may be desirable, depending on the 
thermal imaging system used, that all thermal images first be normalized to a 
standard thermal profile before processing. In alternate embodiments, intended for 
various applications and various environments, these preprocessing steps may 
significantly increase accuracy in recognition or may merely impose unnecessary 
processing overhead. For example, if system 100 is used in connection with an 
outdoor automated teller machine, thermal normalization may be needed to deal with 
seasonally wide variations in surface skin temperature. 

npl-ai1f>d Dpfir-r-iptinn Text (43) : 

As mentioned above, minutiae generator 104 and minutiae data generator 105 are used 
to produce minutiae data signals for a population of known persons. The data 
corresponding to these signals are stored in minutiae database 112. Thermal imaging 
device 102 then obtains a thermal imagf^ of an unknown person 101 and minutiae 
generator 104 produces signals representative of the minutiae and minutiae data 
generator 105 generates data for the minutia for that person. Once these signals 
have been produced, minutiae matcher 106 compares the signals representative of 
person 101 to signals from minutiae database 102 corresponding to minutiae data of 
known persons. In a preferred embodiment, minutiae matcher 106 performs three basic 
functions to obtain a match: alignment of the unknown face, comparison of minutiae 
data, and selection of a match. Each of these functions is described in greater 
detail below. 

Detailed Description Text (45) : 

Because there may not be control over the position of the face of person 101 with 
respect to the field of view of thermal imaging device 102 when image is obtained, 
the orientation of the face may not be such that the facial axes are aligned to be 
horizontal and vertical. Thus, minutiae matcher 106 corrects the orientation by 
rotating the imagf^ such that the facial axes are horizontal and vertical. Next, 
conventional processing using a three dimensional model is applied to correct for 
any rotation or twist of the head. In a preferred embodiment, such processing models 
the head as a sphere with a diameter equal to the apparent width of the face, and 
anti -distorts the imagg^ to provide a view which is normal to a surface plane across 
the forehead and upper lip and in which the enter of the sphere conincides with the 
face center. In a conventional manner, the nose and chin are ignored so as not to 
disrupt positioning of this surface plane. 

Detailed Description Text (47) : 

Comparison of the minutiae data of the unknown person 101 with minutiae data from 
known persons begins by conqparing locations of such minutiae. First, the locations 
of minutiae for a known face are considered, and denoted as M(K)i. Next and allowed 
positional error .epsilon. of one of the M(K)i are ignored. Any M(K)i which are not 
within .epsilon. of on the M(U)j are ignored. This leaves a residual set of minutiae 
pairs. If this set is empty, there is not a match between the two_imag^. Otherwise, 
the characteristics of the corresponding points are compared. 

nf*t:ai1f>d Df*gr-ription Text (52) : 

Further levels of decision requirements can similarly be added to produce the 
desired level of confidence in the match for the application at hand. Each possible 
comparison of the unknown face with known faces is performed, and then the known 
imagf>Pi are rank-ordered according to the goodness to fit (e.g., closeness in metric) 
with the unknown face. 
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Hf^ha-ilRd DfiRrr-i pt i nn Tf^vh (54) : 

Through experience with use of the system on new ■imagpH of known persons, a 
threshold value is established to provide a desired ratio of false positive and 
false negative identifications appropriate to the particular application.' In a 
preferred embodiment, both self -correlations of multiple imagf^g of known persons and 
cross-correlations of different known persons in the dateibase are used to help 
establish this threshold. 

nf>1-a-ilpH Dpffr-r-ip^-i nn Tf>vh (55) : 

If only one known person meets the threshold requirement, that person is selected as 
the match. If no known person meet the threshold requirement, a failure to match 
signal is produced. If multiple known persons match the unknown person to within the 
threshold difference, the best matching person is selected. Alternatively, if 
multiple images of the same known person are referenced in minutiae database 112, 
the person having the highest ratio of matches within the top number of best matches 
may be used. For instance, if there are ten images of each known person in database 
112, the top ten matching images determined by minutiae comparison are considered. 
The person who is associated with the most of the top ten is selected to be the 
matching person. Additional levels of decision requirements may be added, either in 
a simple manner or iteratively, with a determination after each level as to whether 
a match decision can yet be made. 

Detailed Desrr-iption Text (56) : 

In an alternative embodiment, minutiae matching is performed using techniques 
disclosed in U.S. patent application Ser. No. 07/984,514, filed Dec. 2, 1992, and 
U.S. Pat. No. 08/314,729, filed Sep. 29, 1994, which is a continuation of U.S. 
patent application Ser. No. 07/984,514, both of which are hereby incorporated by 
reference in this application as if the entire contents of each had been fully 
reproduced herein. In this alternative embodiment, flash correlation is used to 
match minutiae through a digitized artifact -producing technique. In this embodiment, 
the size of a minutia point is preferably increased to represent the possible error 
in its location, and minutiae are replicated successively along the face axes to 
increase their density and thereby increase the discemability of the correlation 
artifact that indicates a match between tiwo images being compared. Such artifact is 
found to occur if any only if there is a match between two pixelized images . 

Dehaileri nesn-r-i phi on Teyh (57) : 

Other known matching techniques may alternatively be used in minutiae matcher 106, 
with tolerances established for errors due to imperfect knowledge of head position 
or distance, errors introduced by considering the head or face as a two dimensional 
surface or as a sphere, and other systematic and random residual errors. Some known 
fingerprint matching techniques may also be adapted to use with minutiae matcher 
106. By analogizing thermal contour to fingerprint ridges, the multiplicity of 
facial thermal contours may be treated in a manner similar to matching many fingers 
per person. Alternatively, specific areas of the face, such as surrounding the 
canthi, may be selected and used alone for identification. Depending on the 
resolution of the thermal imaging device 102, several hundred minutiae may be 
extracted from a facial thermal image , As noted above, lack of prior knowledge of 
which facial features, and which specific matching techniques are used for any 
particular application by system 100 increases the security of system 10 0 against 
being compromised by third parties. 

Detailed Description Text (59) : 

Another approach to classification of facial thermograms relates to obvious 
characteristics for use in verbally describing a given facial thermogram. Such 
characteristics include whether the canthi are merged or separated; whether the 
thermal contour of the nose is relatively cold, hot, or normal; whether the nose is 
trapezoidal in shape or irregular in shape; the degree of thermal symmetry of the 
forehead; and the degree of symmetry in location of thermal features in the mouth 
comers, the inner curves of the cheeks, the nose, the canthi, and the outer comers 
of the eyes. To be useful, such designations should remain consistent over 
variations in imaging equipment, environmental conditions, physiological variables, 
and other sources of errors. Accordingly, classification should not rely on features 
determined to be highly sensitive to such factors. Classifications based on overall 
image, e.g., those based on some of the distances between the features discussed in 
connection with FIG. 4 may be suitable for use. 

Deha-ileH nesrripf -inn Tey^ (60) : 

Another approach is to use wavelet coefficients that produce the minimum difference 
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between an interpolated wavelet -compressed imagf> and the original image. Depending 
on how many classes are desired, that number of wavelet coefficient sets can be 
generated. Each imagf^ to be classified is compressed and then restored using each of 
the sets. The -imagf* is assigned to the class corresponding to the set of wavelet 
coefficients which best restores the imagf^ to its original form. 

nAhai1f>H nfiRrr-iph-irtn T^yh (64) : 

The usefulness of facial thermal imaging in recognition applications is increased by 
appropriately encoding thermal facial -iniagpFi so that consistent codes are generated 
each time a facial thermogram of a person is obtained. Such a coding scheme reduces 
database search and minutiae matching overhead, thereby allowing faster processing 
using less expensive equipment. In a preferred embodiment, overlaying a grid on a 
face such that 144 cells cover the area of the face, and assigning a binary code to 
each cell, such t hat the cell is encoded with a "1" if the cell contains one or 
more minutiae and "0" if the cell does not contain any minutiae, is found in 
practice to yield good results. Since this encoding scheme preserves the relative 
location of each bit, it is straightforward to ignore selected bits in cases where 
only a portion of a face is imagf^d, due to obstruction, disguise, or orientation. 

Df^t-ailed De5crip^.i rtn Te>yt (66) : 

Although the discussion above has been directed to thermal images of faces, it 
should be recognized that similar techniques and systems may readily be applied to 
■imagRg of Other body parts in accordance with the present invention. It should also 
be recognized that numerous other imaging modalities besides thermal imaging may be 
employed in accordance with the present invention, for example x-ray, NMR, MRI, and 
CAT scan imaging. It should also be recognized that known schemes for pattern 
recognition and graph matching may be applied readily in accordance with the present 
invention, depending on the needs of a particular application. 

Df^ha-ilf^d nf>srr-ip^i nn Tpy^ (68) : 

The method and apparatus described above for facial minutiae extraction can be used 
to develop a standardized minutiae co-ordinate system for identification of medical 
patients and for diagnosis of medical conditions. Because an infrared camera 
operates at a distance from the patient and detects and records only radiant heat 
spontaneously emitted from the body surface, it constitutes a painless, 
non-invasive, passive method of recording patterns of body surface temperatures. 
These patterns have been found to depend upon the underlying vascular structure and 
are unique for each person. Infrared identification therefore provides a method for 
uniquely identifying individuals under all lighting conditions, including total 
darkness. It is not prone to forgery or multiple identity deception and so provides 
convenient and highly secure identification of individuals. The method for 
generating repeatable registration points on the skin surface of the human body 
utilizes discrete minutiae points obtained from the thermal images. Visual 
characteristics of the body, such as size and shape and relative position of body 
parts, are maintained in the infrared image.. In addition, the details of the 
vascular system are indicated by the distribution of temperature cross the skin 
surface. Current infrared cameras are sufficiently sensitive to temperature 
variations that they clearly distinguish the skin directly overlaying blood vessels 
due to the thermal difference caused by the flow of warm blood. The vascular 
structure appears as a white (hot) overlay of the circulatory structure on top of a 
grey scale image of the thermal map of the body, as shown in FIG. 7. 

Derailed Deariript ion Text (69) : 

In FIG. 8, there is shown apparatus 100a for processing infrared i mageH to yield 
repeatable minutiae points corresponding to specific vascular locations under the 
skin. The apparatus includes a thermal imaging device 102 for producing a thermal 
image I. A minutiae generator 104 and minutia data generator 105 are part of the 
program memory 108 as is a minutiae overlay device 116. The set of minutiae obtained 
from any extended are of the body is unique to each individual. In particular, 
facial minutiae are unique between identical twins. The same thermal minutiae are 
repeatedly extracted from a given individual. They are overlaid by the overlay 
device 116 and annotated by an image processor 118 on the infrared image or on a 
visual or any -image obtained from another medical sensor having the same orientation 
to the subject. From the processor, the annotated image signal is delivered to an 
output display /processing device which produces the thermal image with overlay 10. 

Deha-iled neflrr-i p^^r^n Text- (70): 

In FIG. 9, there is shown a modified apparatus 100b to that of FIG. 8, wherein a 
reference minutiae identifier 120 is provided between the minutiae data generator 
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105 and the minutiae overlay device 116 to identify and specify reference points 151 
in the thermal imag^ and overlay 10. The reference points allow manual or automated 
comparison, merging, or registration among a set of images taken at different times 
with different orientations or different medical instr\iments . FIGS. lOa-c illustrate 
minutiae automatically extracted from a facial thermogram as the head turns. Current 
infrared cameras commonly produce 30 frames of video output per second, and minutiae 
extraction and annotation can be performed in real time as those frames are 
generated. 

^>P^a^1pr^ DePirri phi nn Tpvh (71): 

FIG. 11 illustrates alternate apparatus 100c for generating a medical image MI of a 
portion of the body via a medical imaging device 122 such as an x-ray machine. The 
medical imagg^ can be annotated with a minutiae overlay to generate a medical -jmagfi 
with minutiae overlay MIO from the output display/processing device. 

Detailed Description Text (72): 

The inventive technique differs from visible recognition approaches in that it does 
not merely sample a finite nxamber of points on an image; it extracts points which 
have particular meaning. This provides increased resolution at the same time it 
reduces the degree of computation required. The essential features of the technique 
are the uniqueness and invariance of thermograms, the use of a passive imaging 
technique to obtain subsurface details, the use of automated minutiae extraction to 
match different -imagf^R taken of the same individual, and the use of standardized 
minutiae locations to compare different persons or the same person as he grows from 
childhood to adulthood. The matching technique involves developing sets of 
corresponding points in two -tmagf^R, mnrphing one image into the reference, and 
measuring the degree of mnT-ph-i ng as an indicator of the amount of difference. 
Similar automated procedures are then used to verify that imagery is from the same 
pahi p^nt, identify a pa^^^n^ by comparison to a database of imagf^.g, ajid compare 
images taken at different time and/or with different sensor modalities. 

Det-ailefi DeRrr-i i nn Tey^ (75): 

Verification that two medical_ima£f£a are from the same person can be an end goal in 
itself or the first step in further processing the two images to extract comparison 
data. Telemedicine applications, electronic filing systems, insurance claims 
processing, updating of medical records, and extraction of medical histories during 
emergency treatment are some of the situations in which it is essential to have a 
reliable, fool-proof method for positive identification of the subject and precise 
localization of imaged areas. 

Det-aileH neRn-ri phi on Text, (76) : 

The use of infrared identification (IRID) has several advantages over other methods 
for recognition of persons. IRID operates regardless of lighting conditions. It 
requires only a single frame of imagery, taken in 1/30 of a second, for positive 
identification, and so can provide on-the-fly recognition during emergency 
admissions or evacuations. This imagery can be collected at a distance, without 
causing a delay or inconvenience to the siibject. No parts of the system come into 
contact with the subject. Since thermal images are essentially immixne from 
variations caused by illumination and shadows, it is not necessary to control those 
variables. IRID provides continuous identification and confirmation verification of 
ID even in the dark. A cheaper, cruder form of thermal imaging can be obtained from 
use of heat -sensitive crystal sheets, which require contact with the skin. More 
expensive, active imaging of the vascular system can be obtained from laser doppler. 
Either of these could be the source imager for extractions of minutiae; however, the 
operational advantages of passive thermal imagers make them the preferred sensor. 

Detailed Dp.scription Text (77) : 

Thermal minutiae can be obtained from commercially- available thermal imaging devices 
sensitive in the 3 to 5 or 8 to 12 micron wavelength bands. Tmagefi are this type are 
shown in FIGS. 7, 10, 12, and 13. 

Deha-iled DeRrr-ri pf -i nn Tevt (78): 

Current infrared cameras produce a standard analog or digital output providing 3 0 
frames per minute as shown in FIGS. 7-10. Tracking the minutiae from frame to frame 
assists in the exploitation of the dynamic IR imagery by allowing measurements to be 
made over time from the same body locations while accommodating changes in position 
due to respiration, voluntary or involuntary movements of the s\ibject, and 
intentional or accidental variation in the position of the imaging system. The use 
of infrared video imagery also allows the imagery to be recorded in real time for 
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later analysis, and provides a self -documenting chain of custody identification of 
the person recorded, all without the necessity for the cooperation of the person 
being Imaged. 

Dfiha-ilRH np^grr-i ph-i nn Tpyf. (79) : 

Infrared imaging can be used to locate minutiae points over the entire body surface 
which correspond to intersection points and branch points of the underlaying blood 
vessels. This provides a built-in set of registration points on the body's surface, 
which can be annotated onto imagf^R produced by any medical sensor used in 
conjunction with the thermal imager . The registration points then can be used to 
compare and combine medical images taken with different equipment at different times 
and under different conditions, facilitating comparison of those images. Also, the 
minutiae points provide reference points for continuous re -alignment of surgical 
instruments, radiation sources, and other diagnostic or treatment equipment. Since 
the infrared camera is totally passive, it can be used continuously during other 
medical procedures to overlay precise registration points on the other images while 
also monitoring for overheating, shock, hyptothermia, renal failure, and other 
medical conditions. At the same time, the pattern of minutiae points superimposed on 
each image provides positive identification of the patient. Such applications are of 
particular importance during telemedicine procedures. 

neha-ilpri nf^finriptinn Tevt (80) : 

The normal body is basically thermally bilaterally symmetric. Side to side 
variations are typically less than 0.25 degrees Celsius. This fact is used in 
assigning axes to the body ' s^image . Where the skin surface is unbroken, there is 
gradual variation of temperatures across blood vessels, with the highest 
temperatures across the body surface being directly on top of major blood vessels. 
Major thermal discontinuities occur at entrances to body cavities such as the eye 
sockets, nostrils, or mouth. These provide global reference points for automatic 
orientation of the thermal image. Local and relatively minor discontinuities in the 
skin surface occur at scars, moles, bums, and areas of infection. The thermal 
surface can be distorted through pressures and activities such as eating, 
exercising, wearing the tight hats cind other clothing, sinus inflammation, 
infection, weight gain and loss, and body position. However, the minutiae points 
remain constant with respect to their position relative to the underlying blood 
vessels. 

Det:ailed Degcriptinn Teyt. (81) : 

The technique for thermal minutiae extraction and matching can be summarized as 
follows: 1. Current thermal iniagf^ is digitized. 2. Current image is divided into 
pixels, where the size of the pixel relates to the resolution of quality of the 
result desired. 3. Certain pixels are selected as minutiae points. 4. Each minutia 
is assigned a vector having magnitude and directional information in relation to the 
surrounding characteristics of the thermal image. Additional characteristics, such 
as type of minutia may also be recorded for each. Typically for each whole body 
thermal image , there would be on the order of 1200 minutiae. 5. Set of minutiae 
vectors of the current image are compared by computer to the set of vectors of other 
images - 6. Comparison results are used to determine corresponding minutiae from the 
two images, and to morph or mathematically adjust one image with respect to the 
other to facilitate comparison. 7. Differences between the current image and 
database images are computed for either the entire image or for areas of interest. 

ne^a^^erf Desr ri pt. i on Teyt (82) : 

It is desirable that all thermal images in a database be normalized to a standard 
thermal range and be scaled to a standard size during search and comparison 
procedures. Both normalization and scaling eliminate some minute amount of 
identifying characteristics of a particular person or his condition. However, the 
standardization procedures greatly aid in the exploitation of the database by 
reducing the need to calibrate every imaging sensor used to produce images which 
will be filed in, or compared to database images. For example, in accident triage 
with no accurate ground truth reference in the scene and possibly use of inferior 
quality imagers, standardization to constant size and thermal range is appropriate 
in order to match against database. Furthermore, standardization facilitates use of 
simulated imagery for telemedicine and telesurgery applications. For example, when 
incorporated into the military's automated battlefield medical pod, real time 
normalized thermal minutiae can be used to properly position injections and 
application of external pressure to stop bleeding. 

nehaile d ^esrri p^^nn Tey^ (83) : 
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In addition, standardizing database -imagf^fi facilitates comparison of imagery during 
growth from childhood to adulthood, compilation of medical libraries of imagefi from 
large number of people, and automated comparisons of current imagery against the 
base libraries for diagnostic purposes. Standardization to a common use of thermal 
minutiae would also provide a common reference for comparing imagftfi obtained from 
different sensors which produce different resolution images. 

DPl-a-ilpH nf^gr-r-iph-inn Te^.Tct. {86) : 

Blood vessels that carry nutrients to the skin are destroyed when tissue in burned. 
A high powered laser can be used to remove the buzned skin, leaving the healthy skin 
intact. Laser light is differentially absorbed and reflected by live and dead skin, 
such as in the area of a bum. Certain dyes such as indocyanine fluoresce when in 
contact with laser light and can be injected into a patient's blood to indicate 
healthy tissue. Alternately, an IR imager can be used to indicate dead skin vs. 
healthy skin, since the dead skin appears relatively cold as a result of having no 
functioning blood vessels. More than 100,000 person per year in the US alone suffer 
severe bums. The total cost of treating these patients exceeds $2 billion. 
Treatment includes massive transfusions to replace the blood lost during surgery to 
remove dead skin. Blood loss is the main cause of death in bum patients. Continued 
heavy bleeding often prevents a successful skin graft after the burned skin is 
removed . 

DpI-ailpH Dp.cir'-r-iph-inn Tpyh (88) : 

Referring now to FIG. 14, there is shown an apparatus for maintaining the position 
of a surgical instrument 160 relative to a surgical site in a patient 162 during a 
surgical procedure. A thermal imaging device 102 such as an infrared camera, 
generates a thermal -imagp output to a minutiae generator 104 to identify minutiae in 
the vicinity of the surgical site. As described above with reference to FIG. 9, a 
reference minutia pattern is also generated for a statutory patient. A surgical 
instrument position detector 166 senses the position of the instrument and produces 
an output signal corresponding therewith. The minutiae pattern for the patient and 
the surgical instrument positions signals are delivered to a comparator 168. The 
relative position of the instrument to the surgical site is set as a reference at 
time 0 at 170. Displacement of the patient and/or the instmment with respect to the 
reference at times other than 0 is detected by the comparator which can produce an 
output 172 used to reposition the instrument via a device 174 for proper orientation 
to the surgical site. 

np^a^1pr^ nparT-i p^ t on Tpvh ( 92 ) : 

Within the next 10 years, both military and civilian medicine are expected to make 
routine use of telesurgery where the patient and surgeon are not co-located. It is 
estimated that 90% of the information a physician needs to know aibout a patient can 
be acquired and brought to him electronically. In laparoscopic surgery today, a 
surgeon looks only at video images without looking at the real organs at all . 
Dermatology and pathology are both already using electronic images as well as x-rays 
and medical records. Laparoscopic surgery is an electronic form of surgery. 
Teleradiology, telepathology, and teleconsulation are already widely accepted 
electronic medical practices. 

Detailed DfiRcription Text (95) : 

Many drugs, including cocaine and alcohol, are vasoconstrictive siibstances which 
cause cooling of the skin surface, the resultant cooling is detected through passive 
imaging of the thermal energy emitted form the face. In FIG. 15 is shown the thermal 
image of an individual who is siibstance free and in FIG. 16 is a thermal image of 
the same individual after the ingestion of alcohol. 

np^a^1p<i Dfqgnr-ipt-i on Tf^xt. (98) : 

Variations in temperature across the facial surface can be imaged by thermal cameras 
sensitive to wavelengths in the 3-5, 8-12, or 2-15 micron ranges. Current cameras 
can provide thermal resolution better than 0.07. degree. C. and spatial resolution of 
better than 0.02", resulting in 65,000 to 265,000 discrete thermal measurements 
across the surface of the face. For most such cameras, that thermal map is generated 
30 times per second to produce either a standard video output which can then be 
recorded and processed on standard videotape equipment, or a direct digital signal 
which can be immediately input to a computer. 

Detailed neficription Text (104) : 

The technology of this patent involves clinical studies in which known amounts of 
controlled substances are administered to subjects whose thermal images are 
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monitored and stored to provide archival references . Analysis of those images is 
used to determine the time-varying thermal effect of specific drugs at specific 
minutiae sites on the face, which are specified by anatomical landmarks. After an 
individual ingests a drug, changes in his or her thermal signature gradually occur 
until a thermal "climax" is reached after \irfiich the signature gradually returns to 
its normal state. In chronic drug users, permanent physiological changes may occur 
such that there is no longer a smooth total decay of the apparent drug -induced 
effects. By processing a significant number of thermal Imagftfi, thermal signature 
markers are identified and related to standardized vascular system locations whose 
thermal variation are highly correlated with use of the particular substances. 
References may be developed for an individual, for a class of individuals grouped by 
age or other characteristic, or for a general population. 

DRt-a-ilRfi Deacr-iph-inn Tevt- (105) : 

A general determination of substance -free vs. substance -influenced classification 
may be based upon data collected on the thermal effects of various substances of 
interest. In addition, certain sxibstances produce characteristic results which may 
be identifiable from detailed analysis of the thermal signatures associated with 
facial minutiae, and/or with distribution statistics from those signatures. 
Furthermore, the rate of change at any point in time may be discriminator between 
chronic and recent use of each drug. Techniques for processing sequences of thermal 
■imagf^fi may enhance the visibility of bilateral asymmetries, anomalous static 
conditions, and unusual time-varying trends in the thermal signatures associated 
with specific minutiae locations in the face to indicate activity levels of specific 
arteries in the brain which are known to be affected by particular drugs. Therefore, 
when a substance is known to affect particular fiinctions, vascular pathways to the 
corresponding brain areas should be analyzed for related thermal signatures at 
minutiae points along the pathways, under the assumption that increased activity at 
the brain site will be found to correlate with increased vasomotor activity along 
pathways leading to that site, as evidenced by thermal changes. 

nf^^a-i1f>d nRRnr-iph-ion Tf^xh (106) : 

In order to best compare_imag£a from different people and under different 
conditions, facial thermograms must be standardized and registered to common 
coordinates. The preferred approach is to use the standardized infrared minutiae 
coordinate system (SIMCOS) technique which locates standard minutiae points on each 
facial thermogram. In its preferred embodiment, the SIMCOS minutiae correspond to 
anastomoses which are connections or branchings of major superficial veins and 
arteries in the facial area. Approximately 175 such points exist in the face. A 
sxibset of the SIMOS minutiae which relate to blood vessel or areas of the face 
affected by a particular sxibstances is selected. The substance activist at the brain 
site will be found to correlate with increased vasomotor activity along pathways 
leading to the site, as evidenced by thermal changes. 

Dfiha-ilR H r>p.ctr-r-i phi r>n Tpv^ (108) : 

In slibsequent screening of a known individual for a particular substance, his 
current thermal image (r) are analyzed to extract substance -active minutiae which can 
be seen in the availcible image(s) . The set of thermal signatures is compared to the 
substance marker collection and substance -free marker collections. Measures of 
similarity are calculated for individual, class, or general population comparisons 
for each substance of interest. Various correlation associated with substance use, 
or with substance -free references in the system library. Normalized temperatures vs. 
time waveforms for each minutiae can be compared with the corresponding waveform 
from the reference. The waveforms are slid along the time axis until the best fit is 
found, since it is not known when, if at all, the person being screened may have 
used that substance. Another correlation approach involves sampling the thermal 
waveforms and producing a matrix of values, where one dimension of the matrix is the 
number of minutiae used, and the other is the number of temperature samples over 
time. The reference library can include wider matrices, involving longer time period 
than is practical for an operational screening system. The comparison between the 
collected matrix and the reference matrix would use a digital shifting and 
difference calculation to find the best area of match. 

DptailpH Dpptr-riphi nn Tpyh (114) : 

The imagp in the frame grabber and buffer is processed by the face locator 20 which 
determines that the imagg^ includes a single face which is in focus and of a suitable 
size and position. If the image is not suitable according to software criteria 
established within the face locator, a new imagp frame is grabbed and the process 
repeated until a suitable imag e is obtained. 
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Df^ha-ilpd npfir*T- iphinn T^xh (115) : 

The imagf^ is then processed by the minutiae extractor and identifier 22 which 
locates the SIMCOS minutiae points and extracts their positions on the image and the 
corresponding apparent temperatures. Additional frames are grabbed and processed for 
a period of time selected by the system manager. 

Df^tailf^d nRRrr-iphirtn Tt^vt (121) : 

The substance use identification and detection system requires databases of thermal 
signatures from svibstance-f ree and substance- related images. During clinical trials 
in which sxibstances are administered under rigorous protocols, the same apparatus 
may be used to generate the reference databases. 

Detailed DRficription Text (122) : 

A clinical trials investigator will serve as system manger. He will perform set-up 
and initialization of the system, the system manager interface 2, which includes a 
monitor or other display, keyboard, and possibly a printer and other peripherals 
such as a mouse which are normally associated with personal computers. The system 
manager must confirm the identity of the enrollee 2, and input the associated 
identification information into the person identifier and condition identifier 
database 6 within the system processor 8. Since the same person may be enrolled 
several times under various protocols, he is termed the "subject" to emphasize that 
he may have several separate files within the enrollment database 26. The s\abject*s 
current and past medical history data is input to the system the first time he is 
imaged. During each different test involving different substances and/or protocols, 
the specifies of the protocol used, as well as recent medical data, including the 
results of urinalysis or blood tests to detect substance use, will be included in 
the person identifier and condition ID Buffer 6. The contents of that buffer are 
transmitted and stored with the results of the thermal signature extractor 24. 

Detailed Description Text (125): 

After the end of clinical trials, frames are extracted from the archival videotapes 
and used in place of^imagea directly taken by the infrared camera. Each extracted 
frame is processed by the system, and the thermal signature comparator and 
statistical analyzer 36 selects the best matching composite signature and classifies 
the frame accordingly. First, the thermal signature collection of that frame is 
classified as better matching the composite for all substrate -free trials or the 
composite for all substance -related trials. Second, if the subject in the frame is 
judged to be substance -free, the best matching class of database 30 is then 
selected. If the subject is judged to be substance- related, then the best matching 
substance of database 32 is selected. Third, in the substance -free case, the system 
identifies the best -matching enrolled person's signature within the class selected. 
In the substance-related case, the system identifies the best-matching protocol for 
the substance selected. Since the true classification of each subject on the video 
tape is known, the performance of the system in an automated mode can thus be 
determined, and the error rates associated with the classification can be 
calculated. Improvements to the system can then be made according to standard 
techniques for statistical analysis, including the use of neural nets to adjust 
weightings consideration of the data from certain minutiae more or less than others, 
normalizing the thermal variations or not, and normalizing time intervals based upon 
the size or metabolism of each subject. 

Dpt-a-ilpd nf^F!r!•r^p^^nn Tf^.Tct. (128) : 

The imagf^ in the frame greibber and buffer is processed by the face location 20 which 
determines that the image includes a single face which is in focus and of a suitable 
size and position. If the imagf^ is not suitable according to software criteria 
established within the face locator, a new image frame is grabbed and the process 
repeated until a suitable image is obtained. 

Detailed Description Text (129) : 

The -imagf^ is then processed by the minutiae extractor and identifier 22 which 
locates the SIMCOS minutiae points and extracts their positions on the image and the 
corresponding apparent temperatures. Additional frames are grabbed and processed for 
a period of time set into the system by the system manager. 

nPha-ilpH npflr-riph-inn Tpvh (133): 

Various standard methods for locating faces in an imagf* frame can be used. A 
particularly useful approach uses an ellipse detector to find relatively warm 
ellipses (thermal faces) within a relatively cool background. The ellipse is located 
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within the expected height range for humans, the detected temperature is within the 
expected range for human faces, and characteristics common to all facial thermograms 
(hot canthi regioes, symmetry of the eyes, nostrils, ears, cheeks, etc.) are not 
violated. Each detected face in each frame is evaluated to determine if it meets the 
quality requirements for further processing. Requirements include the facial imagg* 
being in focus, being large enough to provide adequate resolution of the facial 
minutiae, being oriented close enough to full face forward and being free enough of 
blockages including beards, eyeglasses, and intervening obstructions, such that a 
sufficient number of facial minutiae can be extracted from the facial image. The 
specific requirements are dependent upon what substances are to be detected, in how 
small a dose, and after what period of item. Faces which do not meet the quality 
requirement are not further considered. Those which are qualified are assigned 
unique tags. 

Detail fid Dftflcription Text (135) : 

The system is designed for stand-alone operation. IT is deployable for programmable 
periods of time, during which it will analyze and classify each face which appears 
within its field of view. The system will not routinely record or store the thermal 
■imagpfi, although provisions are made to do that during testing and evaluation of the 
system in order to allow for improvements to be made in the system and compared with 
earlier results. The output from the system will be graphical results such as shown 
in FIG. 19. The cumulative detection indpy of the y-axis represents the number of 
people who the system estimates have used marijuana, cocaine, or heroin in an amount 
and within a timeframe which results in a residual level indicated by the x-axis 
value at the time of the analysis. The x-axis represents the confidence level of 
drug signature indication, which is related to the detection precision of the 
testing and analysis procedures. Separate curves indicate the specific drugs 
detected, and a composite survey indicates detection of any of the substances. Due 
to the frequent use of combinations of drugs, the composite curve is expected to be 
more significant than its components. 

Detailed Description Text (142) : 

The analyzer 64 of the composite thermal signature compares each imaged person's 
composite thermal signature to a database of signatures associated with known 
substances and protocols, or with known subpopulations such as cocaine addicts or 
alcoholics. More particularly, a signature from a thermal signature database from 
deployment 66 is compared with a signature from the reference thermal signature 68 
is a comparator 70, the output of which is delivered to a thermal signature 
classifier 72. The analyzer output is delivered to a statistical analyzer 74 for 
processing the thermal signatures by batch to produce an output summary to the 
output report generator 76 for the output device 78 . A statistical sufficiency 
analyzer 80 evaluates the report and determines whether the analysis of the 
population up to that point is statistically sufficient for the intended purpose. If 
not, additional frames are analyzed. If the analysis is sufficient, then the system 
is re-initialized by the initializer 50 starting a new collection of faces and 
leading to a new statistical evaluation. 

CLAIMS : 

1. A non-invasive automated method for continuously controlling the re-aligning the 
position of an instrument relative to specific locations on a patient's body during 
a medical procedure, comprising the steps of: (a) generating thermal imagc^R of the 
patient using a non- invasive imaging device spaced from the patient prior to and 
during use of the instrument on the patient; (b) using a minutiae generator to 
identify minutiae in the vicinity of a specific location of interest on the patient 
from said thermal image , said minutiae corresponding with specific branch points of 
blood vessels of the patient; (c) generating a reference minutiae pattern for a 
stationary patient; (d) generating a reference position of the instrument with 
respect to said reference minutiae pattern; (e) detecting deviation from said 
reference minutiae pattern owing to movement of the patient and from said instrument 
reference position owing to displacement of the instrument; and (f ) continuously 
repositioning the instrument with respect to said reference minutiae pattern to 
accurately position the instrument for the medical procedure. 

2. Non-invasive apparatus for continuously controlling and re-aligning the position 
of an instrument relative to specific locations on a patient's body during a medical 
procedure, comprising: (a) non-invasive imaging means spaced from the patient for 
continuously generating thermal imagf^g of the patient prior to and during use of the 
instrument on the patient; (b) minutiae generator means for identifying minutiae in 
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the vicinity of a specific location of interest on the patient from said thermal 
imagp ^ said minutiae corresponding with specific branch points of blood vessels of 
the patient; (c) means for generating a reference minutiae pattern for a stationary 
patient; (d) means for generating a reference position of the instrument with 
respect to said reference minutiae pattern; (e) means for detecting deviation from 
said reference minutiae pattern owing to movement of the patient and from said 
instrument reference position owing to displacement of the instrument; and (f) means 
for continuously repositioning the instrument with respect to said reference 
minutiae pattern to accurately position the instrument for the medical procedure. 
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interpoIat$4 and 13 : 10:38 
textur$3 

SI 802 U USPT,PGPB,JPAB,EPAB,DWPI,TDBDanimat$4nearl5 2003-07-28 

interpolat$4 13:08:28 

S1801 U USPT,PGPB,JPAB,EPAB,DWPI,TDBD animat$4 with 2003-07-28 

interpolat$4 13:08:04 _ 

SI 800 U - USPT,PGPB,JPAB,EPAB,DWPI,TDBD animat$4 same 2003-07-28 

interpolat$4 12:50:43 



4 of 6 



7/29/03 5:05 P^ 



SI 799 U 



SI 798 U 



SI 797 U 



SI 796 U 



SI 795 U 



SI 794 U 



SI 793 U 



SI 792 U 



S1791 U 



SI 790 U 



SI 789 U 



S1788 U 



USPT,PGPB,JPAB,EPAB,DWPI,TDBDwatermark$3 2003-07-28 

nearlO (dct nearlO 04:59:45 
( frequency near4 
domain)) 

USPT,PGPB,JPAB,EPAB,DWPIJDBD watermark$3 2003-07-28 

nearlO (dct near 15 04:59:34 
( frequency near4 
domain)) 

USPT,PGPB,JPAB,EPAB,DWPI,TDBDwatermark$3 2003-07-28 

nearlO (dct same ( 04:59:05 
frequency near4 
domain)) 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD watermark$3 2003-07-28 

nearlO (dct and 04:58:18 
frequency near4 
domain) 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD watermark$3 2003-07-28 

nearl5(dctand 04:58:07 
frequency near4 
domain) 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD (watermark$3 same 2003-07-28 

(dct and frequency 04 : 56: 1 2 
near4 domain) ) and 
moskowitz 

USPT,PGPB,JPAB,EPAB,DWPI JDBD (watermark$3 same 2003-07-28 

(dct and frequency 04:54:45 
near4 domain) ) and 
wie 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD (watermark$3 same 2003-07-28 

(dct and frequency 04: 54 :3 1 
near4 domain) ) and 
narayanaswami 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD watermark$3 same 2003-07-28 

(dct and frequency 04 : 54: 1 1 
near4 domain) 

USPT,PGPB,JPAB,EPAB,DWPIJDBD watermark$3 same 2003-07-28 

(dct or frequency 04 : 5 3 : 5 6 
near4 domain) 

USPT,PGPB,JPAB,EPAB,DWPI,TDBD digimarc and 2003-07-28 

(spatial or domain 04:50:06 
or space or imagery 
or database )and 
imagery and 
database and index 
and embedded and 

- cloud adj cover 

USPT,PGPB,JPAB,EPAB,DWPIJDBD digimarc and 2003-07-28 
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(spatial or domain 


04:49:37 








or space or imagery 










or database )and 










imagery and 










database and index 










and embedded 




S1787 


U 


USPT,PGPB,JPAB,EPAB,DWPI,TDBD digimarc and 


2003-07-28 








(spatial or domain 


04:48:47 








or space or imagery 










or database ) 




S1786 


U 


USPT,PGPB,JPAB,EPAB,DWPI,TDBD 6504571 and 


2003-07-28 








(spatial or domain 


04:46:43 








or space or imagery 










or database ) 




SI 785 


u 


USPT,PGPB,JPAB,EPAB,DWPI,TDBD 6504571 and 


2003-07-28 








(spatial or domain 


04:46:33 








or space orimagery 










or database) 




SI 784 


u 


PGPB 


6504571 and 


2003-07-28 








(spatial or domain 


04:46:03 








or space orimagery) 




S1783 


u 


PGPB 


6504571 and 


2003-07-28 








(spatial or domain 


04:45:01 








or space)and 










imagery 




S1782 


u 


PGPB 


digimarc and 


2003-07-28 








patch$3 and 


01:09:42 








watermark and 










aerial and platform 










and map and 










generating and 










embed$4 and 










image$2 and 










indicators 1 and 










identif$5 and 










locators 1 




S1781 


u 


PGPB 


digimarc and 


2003-07-28 








patch$3 and 


01:08:23 








watermark and 










aerial and platform 










and map and 










generating 
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